WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory

Abstract: We present WorldDirector, a highly controllable video world model framework designed for persistent dynamic object memory and unrestricted viewpoint exploration. Unlike existing world models that entangle physical dynamics with pixel rendering and rely on continuous visual observation to sustain motion, our framework explicitly decouples semantic motion orchestration from visual generation. By leveraging an LLM to coordinate 3D trajectories with camera movements and subsequently employing these orchestrated trajectories as control signals for video generation, our approach ensures strict physical logic and appearance stability, successfully preserving the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view. Experimental results demonstrate that our method supports the synthesis of complex and extended events with unprecedented controllability and persistent dynamic object memory. Project Page: https://worlddirector.github.io/

• The paper introduces a decoupled framework that separates semantic planning from latent video synthesis to ensure persistent dynamic memory.
• It employs LLM-driven 3D trajectory planning and explicit appearance anchoring to maintain object identity across extended occlusions.
• Experimental results demonstrate superior PSNR, SSIM, and DSC metrics, underscoring its effectiveness in long-horizon, dynamic video simulation.

Controllable World Simulation with Persistent Dynamic Memory: An Analysis of "WorldDirector" (2607.02517)

Motivation and Context

The paper "WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory" (2607.02517) addresses critical limitations inherent in contemporary video world models—namely, the inability to sustain object permanence and accurate dynamic object memory over extended sequences, particularly when entities leave the camera's field of view. Previous methods either rely on entangled generative priors, leading to motion collapse and identity errors after long occlusions, or employ explicit tracking, which is computationally infeasible for complex scenarios with multiple dynamic entities. This work introduces a novel methodology for decoupling the challenges of semantic-level motion orchestration from latent video synthesis, thereby providing a scalable, controllable, and memory-robust generative framework for video simulation.

Methodological Overview

The core contribution of WorldDirector lies in its explicit architectural and procedural decoupling:

1. Semantic-Oriented 3D Planning: An LLM (LLM, e.g., Gemini) orchestrates 3D trajectories and camera motions, translating user instructions—including open-ended, text-described world events—into structured plans for both entities and viewpoints.
2. Conditioned Latent Video Generation: These 3D semantic trajectories are projected into 2D bounding box sequences (Location Conditions) that serve as explicit conditioning inputs for a causal latent video model. Simultaneously, Appearance Conditions derived from retrieved RGB object segments anchor visual identities for dynamic entities across frame boundaries and re-entries.
3. Persistent Dynamic Memory: Temporal context is preserved via explicit context retrieval—combining static background overlap and dynamic object coverage—and asymmetric cross-attention masking prevents noisy tokens from contaminating historical memory.
4. Causal Chunk-Based Generation: The video is synthesized autoregressively in temporal "chunks," with dynamic memory and conditioning updated sequentially, enabling essentially infinite-horizon simulation provided adequate computational resources.

   Figure 1: Overview of WorldDirector’s architecture—LLM-driven 3D trajectory planning projected as 2D location conditions and integrated appearance conditions, feeding into causal chunk-based latent video synthesis.

Key Model Architecture and Technical Innovations

• Auxiliary Conditioning Channels: The video generator, based on LingBot-World-Base, extends its latent space with two major channels: location ($\mathcal{B}$, via instance-specific color masks) and appearance ($\mathcal{A}$, based on cross-temporal RGB crops), encoded using a shared 3D VAE. Explicit concatenation of these features in latent space provides deterministic geometric and visual priors.
• Temporal Drop Mechanism: To mitigate overfitting to static appearance references and avoid "sliding artifacts" (i.e., rigid motion extrapolation using appearance anchors), a sparse-sampling bottleneck restricts the density of appearance conditions after initial entry frames, forcing the model to synthesize plausible articulated motion.
• Contextual Memory Integration: Dual-perspective static and dynamic context retrieval, coupled with shifted Rotary Position Embedding (RoPE) for context tokens, ensures each chunk’s synthesis is robustly anchored in prior scene evolution with clean memory isolation.
• Spatial-Aware Prompt Injection: By leveraging entity-specific cross-attention weighted by 2D bounding box locations, fine-grained semantic control is mapped precisely to spatial regions within the video, enabling multi-entity choreography and promptable event scripting.

  Figure 2: Qualitative comparisons with baselines, showing superior subject permanence and controllable dynamics in long-horizon scenes.

Experimental Evaluation

The authors benchmark WorldDirector against a suite of recent video world models, including Yume 1.5, HY-World 1.5, Infinite World, LingBot-World, and HyDRA, using a bespoke 100-sample dataset emphasizing dynamic entity occlusion and reentry.

Quantitative Results

WorldDirector achieves strong performance across all fidelity and consistency metrics, outperforming all baselines in PSNR (18.127 vs. 14.782), SSIM (0.502 vs. 0.455), and LPIPS (0.359 vs. 0.398). On the Dynamic Subject Consistency (DSC) measure, critical for evaluating reappearance fidelity after occlusion, the method also establishes a new state-of-the-art despite higher scene dynamics—a notable result given that several baselines "cheated" this metric by generating less motion.

Qualitative Results

Figure 2, Figure 3, and Figure 4 demonstrate that only WorldDirector correctly synchronizes camera trajectories with object motion, preserves identity after extended disappearance, and enables scene evolution as specified by user prompts. Baselines exhibit motion stasis, identity collapse, or unsynchronized dynamics, even when reference videos or initializations are provided.

Ablation Studies

Figure 5: Ablation on Appearance Condition—removing explicit visual anchoring leads to identity drift in reappearing dynamic entities.

Ablating the Appearance Condition causes severe identity drift after occlusion, confirming the necessity of explicit visual anchors for dynamic memory. Further, removing the dynamic context degrades temporal identity anchoring, while eliminating the temporal drop mechanism induces motion rigidity.

Figure 6: Ablation on Dynamic Context and Temporal Drop—showing failure modes when either is omitted.

Controllability and Promptable World Events

A significant implication is the extensibility to prompt-driven world event creation: the LLM planner can inject entirely new entities and events at arbitrary temporal points, initializing their appearance directly from text. This flexibility is demonstrated in Figure 7.

Figure 7: Generation of promptable world events, highlighting the ability to introduce new entities and scripted interactions within the simulated environment.

Viewpoint Control

Figure 8: Flexible viewpoint management, including third-person panoramic tracking and first-person viewpoint switching within a single scene.

Explicit trajectory and camera control further enable viewpoint reconfiguration—including seamless first/third-person transitions and panoramic sweeps—addressing a key gap in prior models.

Discussion, Implications, and Future Directions

Practical implications are immediate in domains such as synthetic data generation for embodied AI, virtual world prototyping, film and game previsualization, and interactive simulation environments. The model’s modularity and causal chunking strategy afford scalable long-horizon synthesis with robust multi-entity consistency—addressing a persistent failure mode in pure diffusion or transformer-based end-to-end world models.

On the theoretical side, this approach demonstrates that disentangling high-level world dynamics from low-level pixel processes is not only beneficial but essential for persistent world modeling in generative AI. The explicit use of symbolic planning (via LLMs) to guide video synthesis elucidates a compelling hybrid trajectory that may generalize to broader multimodal generative scenarios, including robotics and embodied reasoning.

Limitations include present reliance on synthetic, game-derived data, resulting in visual artifacts (e.g., occasionally unnatural motion or insufficient photorealism). Bridging this simulation-to-reality gap remains a key future direction, as does scaling the LLM-planning interface for open-ended, user-driven world authoring at finer granularity. Integrating stronger 3D geometry and physics fidelity, as well as real-world dataset alignment, will be crucial for deployment in production-quality simulators or for training robust embodied agents.

Conclusion

WorldDirector (2607.02517) represents a methodologically rigorous advancement in video world modeling, introducing explicit semantic planning, deterministic location and appearance anchoring, and causal chunk-based persistent memory. By decoupling control from synthesis and leveraging explicit multimodal context, WorldDirector achieves superior performance in dynamic memory, object permanence, and user-directed controllability, defining a scalable paradigm for future interactive world simulators. Continued development toward higher photorealism and real-data integration is likely to further broaden the impact and applicability of the approach.