World Narrative Model for Highly Controllable Video Generation: A Paradigm Shift from Pixel Sampling to Physical World Orchestration

Abstract: The fundamental obstacle to industrial grade video generation is the lack of controllability: existing models treat video as a pixel distribution sampling problem, bypassing the explicit, instance level $4D$ $(3D + T)$ physical world. Consequently, content creators cannot specify geometry, motion, camera parameters, or lighting in a deterministic, quantitative way, leading to the infamous ''gacha'' loop that makes professional content creation prohibitively inefficient and expensive. To address this, we introduce the World Narrative Model (WNM), a paradigm that decouples what to render -- the structured physical narrative -- from how to render -- the pixel generation process. WNM replaces end-to-end black-box sampling with orchestrated $4D$ pre-visualization for media generation. Collaborative agents translate sparse multimodal inputs, including text, reference videos, and sketches, into a fully editable world representation with scene geometry, object layouts, character/animal skeleton motion, trajectories, camera motion, and lighting at quantitative, physically meaningful granularity. This representation acts as a deterministic structural blueprint that drives existing video foundation models, either frozen or lightly adapted, to render final footage, turning the base model into a faithful neural shader. Built on this engine, our human-AI platform supports automatic world generation and pre-visualization aligned with professional filmmaking pipelines, while director consoles enable seamless human refinement. Experiments show that WNM greatly reduces probabilistic ``gacha'' calls and produces videos whose layout, motion, and cinematography closely follow creator intent. The framework is open and modular, allowing each component, such as world representation, control agents, and adapters, to be independently improved. Project website: https://glassroom.sjtu.edu.cn/WNM/.

• The paper introduces a novel controller-renderer architecture that decouples explicit 4D world parameterization from pixel rendering for enhanced controllability.
• It employs modular agentic workflows to achieve precise, deterministic control over scene layout, asset placement, motion planning, and camera/lighting settings.
• Quantitative evaluations reveal significant gains in scene, motion, and camera control, reducing generation attempts by up to 72% and overall creation time by nearly 60%.

World Narrative Model: A Paradigm Shift in Controllable Video Generation

Motivation: The Limits of End-to-End Video Generative Models

Controllable video generation remains a core unmet requirement in industrial content creation despite rapid advances in visual fidelity and temporal coherence by models such as Sora, Seedance, and Kling. Existing foundation models, which treat video as a high-dimensional conditional sampling problem, exhibit critical shortcomings in real-world usability. These include stochasticity (non-reproducibility), the inability to specify deterministic physical parameters (e.g., spatial relations, motion, lighting), and pervasive entanglement that prevents fine-grained or local edits. Current control mechanisms—including text prompts, reference images/videos, and pose or mask conditioning—fail to provide structural, decoupled, and reproducible authority over content generation, resulting in iterative “gacha” loops that are fundamentally inefficient for professional workflows.

Figure 1: Three key dimensions that indicate the controllability in video generation task, including decoupling, consistency and spatial-temporal completeness.

The authors disambiguate controllability into four orthogonal dimensions—precision, decoupling, consistency, and spatio-temporal completeness—none of which are adequately supported by current architectures. The root cause is architectural: all mainstream models bypass the explicit 4D ($3D + T$) physical world, learning a monolithic pixel distribution without any entity-level geometric or physical abstraction. As illustrated, this leads to a severe mismatch between sparse user intent and the massive pixel space, entangling semantic and physical control at the pixel level.

Figure 2: Motivations of the proposed new video generation paradigm: from end-to-end generation towards two-phase task decoupling, i.e., full-dimensional world control and pixel render.

The World Narrative Model (WNM): Decoupling Control and Rendering

The World Narrative Model introduces a fundamental architectural shift: decoupling video generation into a Controller (physical world orchestrator) and a Renderer (pixel generator). The Controller constructs an explicit, structured, and fully editable 4D world narrative—a scene graph with precise, quantitative specifications for scene geometry, asset placement, character motions and trajectories, camera path, and lighting/shading parameters. All these are parameterized at instance-level granularity, supporting deterministic and modular manipulation. The Renderer, typically an off-the-shelf frozen video foundation model, acts as a neural shader: it consumes the world narrative (as rendered reference frames or via lightweight adapters) to produce photorealistic video, inheriting the determinism and controllability of the world narrative blueprint.

This controller-renderer split addresses the core failure points of previous approaches:

• Explicit entity-level parameterization: All objects, agents, and cameras are represented with physical attributes amenable to direct manipulation and quantitative specification.
• Decoupling and determinism: Edits to one attribute (e.g. lighting) neither entangle nor drift other attributes. Given identical control parameters, rendering is reproducible.
• Seamless human-AI collaboration: The creator can author, preview, and iteratively refine any aspect using structured interfaces, then trigger high-fidelity rendering at will.

Figure 3: An systematic overview of our proposed World Narrative Model based on collaborating agentic workflows: scene layout, asset generation/placement, motion/trajectory, and camera/lighting.

System Architecture: Agentic Workflow Modules

WNM operationalizes this paradigm through a set of modular, collaborative agentic workflows:

Scene Layout Agent

Given text, reference images, or video, a scene layout agent parses narrative and visual cues into a structured, procedural specification (e.g., Blender Python scripts), constructing and refining a 3D scene entity graph. A generator agent produces candidate layouts, while a supervisor agent validates physical constraints and cross-modal alignment, iteratively closing the loop until engineering and visual requirements are satisfied.

Figure 4: Module 1: scene layout generation agentic workflow.

Asset Generation and Placement Agent

Assets are retrieved (from a large-scale library via multi-stage ranking) or generated (using a proprietary octree-latent diffusion model) according to scene graph node specifications. Geometry-aware placement is performed with physics-aware validation and iterative refinement, ensuring physical plausibility and context-sensitive arrangement.

Figure 5: Module 2: asset generation and placement agentic workflow.

Motion and Trajectory Planning Agent

Motion generation is performed by a factorized spatio-temporal transformer—capable of cross-species skeletal layout (human, quadruped, bird, etc.) and trained on diverse 4D motion datasets—with explicit joint-level parameterization. A trajectory planner adapts motion to the constructed 3D scene, using a closed-loop “render-evaluate-reprompt” cycle with VLM supervision to enforce collision avoidance and semantic-narrative consistency.

Figure 6: Module 3: actor motion and trajectory generation agentic workflow.

Camera and Lighting Agent

Camera agents generate narrative-aligned trajectories and lens parameters via retrieval from a cinematography knowledge base, optimized against scene geometry and narrative timelines. Lighting agents similarly retrieve and place physical light sources consistent with desired mood and scene layout, with physically plausible shadow and color temperature synthesis.

Figure 7: Module 4: cinematography and lighting setting agentic workflow.

Human-AI Collaborative Interfaces

A unified creation platform exposes workflow canvas, entity node graph, and narrative timeline paradigms, all synchronized and fully editable. Four dedicated control panels (scene/asset, motion/trajectory, cinematography, lighting/atmosphere) provide both automatic generation (from text or references) and precision editing, supporting industry-standard parameter formats and direct manipulation.

Figure 8: A diagram visualization of four director's control panels: scene, asset, motion, and camera manipulations.

Quantitative and Qualitative Evaluation

WNM is benchmarked head-to-head against both pure prompt-driven (“novice mode”) and omni-reference (“professional mode”) pipelines using extensive testbeds in diverse cinematic scenarios. Good/Same/Bad (GSB) scoring by professional creators reveals a consistent and significant advantage for WNM across all controllability dimensions.

• Scene control: WNM achieves superior GSB scores ($3.21$ vs text, $2.23$ vs omni-reference), demonstrating precise placement, improved temporal consistency, and robust cross-frame geometry.
• Motion/trajectory: Humanoid and multi-agent coordination, joint-level articulation, and trajectory following are all substantially improved, particularly for long-horizon or intricate interactions.
• Camera control: WNM achieves GSB scores of $3.33$ and $2.35$ over text and omni-reference baselines, demonstrating tight alignment with specified narrative pacing and framing.

User studies record $66$–$72\%$ reduction in generation attempts and $48$–$60\%$ lower total creation time for both novices and professionals. The highest subjective gains are in controllability and result quality. Qualitative examples highlight robust handling of complex scenarios such as narrow street chases, large-scale coordinated crowds, precise bullet-time camera effects, and challenging multimodal lighting or zoom-ins.

Figure 9: Visualization of six representative video generation results by using WNM as control; top: rendered video, bottom: temporally aligned WNM 3D+T representations.

Limitations and Future Research

Key limitations center on:

• Insufficient agent-level robustness and fine-grained accuracy for some professional production use cases.
• Lack of explicit modeling of human-object interaction.
• The current system relies on direct white-box reference conditioning rather than learned adapters for foundation models.
• Absence of large-scale standardization for world narrative representation formats, control interfaces, and controllability metrics.

Ongoing and future directions include expanding multimodal training data (particularly for interaction and complex narrative scenarios), integrating mathematically explicit geometric computation APIs, benchmarking industrial pipeline compatibility, developing adapter learning for model integration, and leading efforts on open, hierarchical representation standardization for world narratives.

Conclusion

The World Narrative Model framework (2606.31946) represents a significant architectural and methodological advance in the quest for truly controllable video generation. By instantiating a controller-renderer separation, grounded in explicit 3D+T world parameterization, it resolves fundamental limitations of prior “black-box” approaches. Experimental evidence substantiates substantial gains in controllability, efficiency, and user empowerment, setting a new baseline for both academic and industrially scalable media generation methodologies. This approach provides a robust, extensible substrate for future research—including agentic toolchains, representation standards, and collaborative human-AI content pipelines—that will underpin the next generation of professional generative media systems.