Holo-World: Unified Camera, Object and Weather Control for Video World Model

Abstract: Video world models are moving toward preserving an observed world under controllable camera and object motion while allowing its environmental state to change. Yet these controls remain isolated, and weather generation typically relies on a source video or reconstructed scene that already specifies future structure. We study a first-frame-anchored source-to-state setting, where the model starts from a single image and follows explicit camera and object controls and an optional weather instruction, then generates a video that either preserves the source world or transfers it to a target weather state. To address these challenges, we first build HoloStateData, a state video dataset that turns diverse videos into unified control samples for camera, object, and weather supervision. Second, we introduce Holo-World, a unified controllable video world model that jointly controls scene from a single image. Its Unified Scene Adapter factorizes world preservation and weather transfer into distinct parameter subspaces, using rendered background, geometry buffers, and object controls to maintain controlled scene structure while modeling weather-dependent appearance and particle effects. Additionally, Scene-Weather Decomposed CFG guides scene and weather residuals separately, strengthening target weather effects without over-amplifying the full condition. Quantitative and qualitative experiments demonstrate that Holo-World maintains precise camera and object control with consistent scene structure while transferring scenes into diverse target weather state, outperforming video-to-video weather editing baselines on weather-state generation. Our project page is available at \url{https://xiangchenyin.github.io/Holo-World/}.

• The paper introduces a unified architecture that disentangles and controls camera, object, and weather attributes in video generation.
• It leverages a specialized data pipeline (HoloStateData) and dual-path adapters (UniSA and SW-CFG) to ensure precise world preservation and vivid weather editing.
• Experiments demonstrate Holo-World's superior performance in subject consistency, background preservation, and weather alignment compared to leading baselines.

Holo-World: Unified Camera, Object and Weather Control for Controllable Video World Modeling

Introduction and Motivation

Controllable world models for video generation have advanced rapidly, yet high-fidelity manipulation of camera, object, and environmental attributes—especially weather—remains segmented, with most models excelling only along isolated axes. Existing strategies largely focus on either camera/object motion or weather-state generation, but rarely achieve a truly factorized and unified control scheme. Consequently, robust single-image-to-video (I2V) generation that respects source-world consistency while enabling explicit environmental edits is beyond the reach of prior art, most of which rely on full source video input or pre-existing scene reconstructions.

Holo-World addresses this challenge by proposing an architecture and data pipeline supporting explicit, disentangled control over camera, object, and weather in video synthesis, leveraging both architectural disentanglement and a purpose-built data platform, HoloStateData. This approach directly targets two enduring obstacles: the absence of aligned, unified data for the full control space, and the entanglement of appearance preservation with environmental state modification in conditional video generation.

Figure 1: Holo-World achieves unified, explicit control over camera, object, and weather domains within a single video world model.

HoloStateData: Unified, Factorized Data for State Supervision

Modern controllable video generation lacks datasets with aligned camera, object, and weather control supervision. HoloStateData addresses this by structuring video samples as source-to-state pairs, where every data record includes: a starting frame, rendered static-world controls (e.g., RGB appearance, geometry), object controls (bounding boxes), and explicit prompts for both scene (excluding weather) and weather states. This systematic separation prevents leakage between source condition and target effect, required for proper supervision of entangled objectives.

The construction pipeline for HoloStateData is threefold: (i) multi-source data collection—including real-world videos (from SpatialVID-HQ), Unreal Engine-rendered simulated weather pairs, and video-to-video (V2V) weather-transferred synthetic samples; (ii) dense annotation of camera trajectory, object masks, controlling geometry, and separated textual prompts; and (iii) rendering of explicit model-facing controls such as background RGB, depth, normal, and object bounding boxes from only the source side.

Figure 2: The HoloStateData construction pipeline produces source-side controls and paired target-weather videos for supervision, ensuring strict separation between scene evidence and weather-state targets.

This data organization enables both ‘no-edit’ (for world preservation) and weather-transfer training, with the balanced dataset supporting specialized evaluation on 150 benchmark samples (50 per Real, Simulation, and V2V subset), and targeted control over four weather regimes (cloud, rain, snow, fog).

Unified Scene Adapter (UniSA) and Scene-Weather Decomposed CFG

Holo-World’s central architectural contribution is the Unified Scene Adapter (UniSA), which introduces strict parameter-space disjointness between background/scene preservation and weather transfer. UniSA comprises World and State Adapter modules, each tuned on explicit control streams: rendered background and camera/object motion for the World Adapter, appearance anchor and target weather textual prompt for the State Adapter. Residual updates are applied at selected DiT blocks in a frozen Wan2.1-T2V-14B backbone, maintaining a single end-to-end model with factorized control semantics. During training, only adapter parameters are updated via a flow-matching objective, the backbone and VAE remaining immutable.

Figure 3: High-level overview of Holo-World, where factorized source-to-state controls are mapped via distinct residual paths for world preservation and weather-state transformation.

To prevent guidance collapse during inference—where classifier-free guidance (CFG) typically entangles scene and weather effects—Holo-World applies Scene-Weather Decomposed CFG (SW-CFG), computing and scaling residuals for scene and weather independently. This dual-residual approach allows the system to strengthen weather modification without destroying source-world consistency.

Experimental Validation

World Preservation and Weather Transfer: Quantitative and Qualitative Results

Experiments show Holo-World consistently outperforms strong baselines (Uni3C, GEN3C, VerseCrafter, NeoVerse for general world modeling; Cosmos-Transfer2.5, Wan2.7-Edit for weather editing) on both world-preservation (Real) and weather-transfer (Simulation+V2V) tracks. On VBench-I2V, Holo-World delivers the strongest overall (89.05), along with best or second-best performance on subject consistency and background preservation scores. Rotational and translational camera control errors, as well as object motion control, are minimized compared to all baselines.

For weather transfer, despite operating under a stricter I2V contract (only single starting frame input, no future video), Holo-World achieves 86% Weather Alignment and a VLM Evaluation score of 68.51, well above the best video-to-video weather-editing system. Human preference rates are also strongly in favor: 83% versus Cosmos-Transfer2.5 and 62% versus Wan2.7-Edit.

Figure 4: Qualitative results demonstrate world preservation in Real rows and diverse, high-fidelity weather-state transfer (rain, snow, fog, cloud) with source-world consistency in Weather rows.

Ablation studies show additive improvements from geometric buffer controls, UniSA, and SW-CFG. The final model uniquely balances source preservation and strong weather transformation, while vanilla CFG either under-edits weather or destroys background fidelity.

Figure 5: The introduction of UniSA reduces interference between world preservation and weather-state transfer, as visualized by ablation results.

Figure 6: Guidance decoupling—contrasting no CFG, vanilla CFG, and SW-CFG—demonstrates improved weather strength without loss of source-world structure.

Multi-Weather Control and Generalization

Holo-World flexibly supports sampling under various weather prompts with fixed camera/object/scene controls. Diverse weather outputs (e.g., cloud, rain, fog, snow) on identical world evidence confirm the model’s scalability and prompt-responsiveness.

Figure 7: Under fixed camera/object controls, changing only the target weather prompt produces realistic, physically consistent videos in a variety of weather states.

Additional visualizations support that background-consistent generation persists across training regimes, and weather transformations are spatially and temporally coherent, aligned with source-control interfaces.

Figure 8: Visualizations demonstrate background consistency in Real cases and weather-state transformation under controlled generation in Weather cases.

Implications, Limitations, and Future Directions

Holo-World’s explicit, factorized design guides the field toward more interpretable, reliable, and extendable models for controllable video generation. By enforcing strict decoupling between world and state in both data curation and model structure, Holo-World sets a precedent for architectures that can be fine-tuned or extended along new axes of control (e.g., lighting, time-of-day, stylization). Such modularity is essential for future deployment in interactive simulation, virtual environment authoring, or augmentation for RL agents with grounded, editable environments.

On the theoretical front, this approach emphasizes the necessity of special-purpose architectures and datasets for entangled control problems, suggesting a shift away from monolithic conditioning strategies typical of standard diffusion backbones.

Practical limitations remain. Holo-World does not reach the complexity or reactivity required for general-purpose physical simulation; its weather edit space is defined by target prompt taxonomy and the distribution of supervised training examples. The decoupling strategy, while effective, may require further elaboration to capture multi-factor joint editing (e.g., weather and time, weather and lighting).

Conclusion

Holo-World establishes a new benchmark for unified, explicit control of scene viewpoint, dynamic entities, and environmental state in single-image-to-video generation. Through HoloStateData and the dual-path UniSA/SW-CFG design, this model matches or exceeds the accuracy of the best available methods for preservation and weather editing, under strictly weaker input assumptions. Its data and architectural choices offer a template for future multi-factor controllable video modeling, with clear implications for the simulation, content creation, and world modeling communities.