- The paper introduces a unified framework combining fine-grained SMPL-based human motion control and causal camera trajectory control for temporally consistent video generation.
- It employs a Fast-Slow Memory training paradigm and t-guided projection mechanism to mitigate error accumulation and improve condition adaptation over long rollouts.
- Empirical evaluations on a large-scale 20M video dataset demonstrate superior motion smoothness and subject fidelity compared to existing state-of-the-art methods.
Fast Autoregressive Video Generation with Compositional Human-Camera Control
Autoregressive world models in video generation have demonstrated substantial progress in visually realistic and temporally stable video synthesis. However, integrating heterogeneous control signals—specifically, fine-grained human motion and causal camera trajectories—has remained a central technical obstacle. Previous methods frequently suffer from either rapid temporal drift in long-horizon rollouts, poor disentanglement of control modalities, or trade-offs between visual fidelity and controllability. This work introduces a unified approach for decoupled yet compositional control, targeting temporally consistent world simulation where both articulated human motion and camera dynamics are precisely directed.
Model Architecture and Key Innovations
The "Directing the World" framework is architected atop the MMPL backbone (Xiang et al., 5 Aug 2025), which enables block-wise autoregressive generation with a plan-then-populate mechanism to mitigate error accumulation across temporal horizons. The pipeline is organized into two sequential control branches: (1) a fine-grained SMPL-based human motion control path, and (2) a camera trajectory control module, both integrated via ControlNet-style condition injection.
A salient innovation is the Fast-Slow Memory training paradigm: the pretrained generative backbone (slow memory) is largely frozen, with newly introduced control-specific modules (fast memory) being adapted with higher learning rates. This substantially improves the controllable adaptation process, reduces training instability, and minimizes interference among spatiotemporal priors and control heads.
For human motion conditioning, the SMPL representation is spatially aligned to the target subject, encoded into a shared latent representation using the same VAE as the video stream, and injected via a timestep-guided projection (t-guided projection). This mechanism dynamically rescales and adapts the control signal at every denoising step, addressing the issue of condition-latent mismatch and mitigating both motion under- and over-conditioning artifacts. Additionally, a refined Motion Classifier-Free Guidance (Motion-CFG) design utilizes a black video latent rather than a zero tensor for the unconditional branch, achieving improved stability and adherence to SMPL-based control in presence of strong textual priors.
Camera control is managed through a causal-aligned approach: the module encodes the entire planned trajectory using global self-attention but only injects block-local features during autoregressive rollout, preserving causality and synchronization with video synthesis blocks.
Dataset Construction and Training Regime
A large-scale, multi-modal dataset is constructed from 20M iStock videos. Rigorous multi-stage filtering ensures only high-quality video, reliable SMPL motion, and accurate camera trajectories are retained, with automated aesthetic and human-motion scoring. The dataset is partitioned into motion-centric samples (high human motion, static camera) for initial motion control learning and camera-centric samples (dynamic motion and camera) for subsequent joint control adaptation, resulting in ~50k high-quality synchronized examples. The two-stage curriculum allows the human-motion prior to be established before compositional integration of camera trajectory, further reducing modality entanglement.
Empirical Evaluation
Comprehensive experiments benchmark performance across long-horizon generation tasks, using both established datasets (e.g., APRIL-AIGC/Ultra Video-Long) and extensive quantitative and qualitative protocols. The method achieves superior overall control and generation quality compared to strong baselines, including Uni3C (Cao et al., 21 Apr 2025), WanMove (Chu et al., 9 Dec 2025), and various FunCamera/Motion models. The proposed framework exhibits the highest overall composite score under joint control, with improved temporal flicker, motion smoothness, and subject/background consistency across extended durations (15s–2 mins).
Ablation studies highlight that Fast-Slow Memory training delivers consistent improvements in all measurable facets, particularly temporal consistency and subject identity preservation. The t-guided projection mechanism significantly mitigates motion distortion and jitter relative to timestep-agnostic strategies.
Multi-Person and Practical Agent Modeling
The pipeline supports both targeted single-person control within scenes containing multiple humans and simultaneous multi-person motion guidance—capabilities crucial for realistic embodied agent simulation. The system’s ability to harmonize human dynamics and scene-level camera exploration demonstrates a step towards generalizable, interactive world models for multi-agent scenarios.
Limitations and Future Directions
Current limitations include insufficient granularity in controlling fine-scale body articulation (e.g., hand or facial motion), residual error accumulation over very long rollouts (manifested as brightness and color fluctuations), and slightly weaker camera pose accuracy compared to models that leverage explicit scene geometry conditioning. These issues are predominantly constrained by the granularity of SMPL estimation, the scale and diversity of the training set, and architectural trade-offs in condition injection. Theoretical future improvements entail scaling the dataset with higher-resolution human-centric samples, enhancing the granularity and modality of control signals, and incorporating text-based CFG distillation to further stabilize global scene appearance.
Conclusion
"Directing the World" represents an authoritative advance in unified, compositional, and efficient controllable world modeling. By decoupling the learning of human motion and camera control while preserving an autoregressive video prior, the system achieves a robust trade-off between long-term temporal consistency, visual fidelity, and fine-grained control—integral requirements for downstream deployment in embodied AI, simulation, and interactive content creation. The compositionality and training architecture generalize well to multi-agent settings and provide a strong foundation for future research on agent-centric world simulation (2606.27964).