State-Aware Video World Model

• State-aware video world models are generative frameworks that explicitly maintain latent world states for simulating video sequences and causal reasoning.
• They combine explicit methods (e.g., 3D geometry, occupancy grids) with implicit memory mechanisms to ensure persistent and manipulable state representations.
• These models enable enhanced video prediction, interactive simulation, and reinforcement learning by enforcing physical, semantic, and causal constraints.

A state-aware video world model refers to a generative or predictive computational model that explicitly maintains and updates an internal representation of the latent world state while simulating video sequences, frequently conditioned on agent actions or controls. This approach moves beyond traditional autoregressive video prediction—where outputs are driven by short-term frame context—by enforcing that a structured state variable encodes persistent, manipulable, and causally-consistent aspects of the world. This article surveys foundational formulations, architectural trends, explicit and implicit state management strategies, alignment with physical and semantic constraints, and major benchmarks in state-aware video world modeling.

1. Formal Problem Setting and State Representation

A state-aware video world model formalizes the generative process over observations (frames) $x_t \in \mathcal{X}$, latent world states $s_t \in \mathcal{S}$, and action/control variables $a_t \in \mathcal{A}$ via an explicit Markovian factorization:

• Observation decoder: $g: \mathcal{S} \to \mathcal{X}$, $x_t = g(s_t)$ maps latent state to pixels or video frames.
• Dynamics model: $f: \mathcal{S} \times \mathcal{A} \to \mathcal{S}$, $s_{t+1} = f(s_t, a_t; \theta)$ propagates the internal state given actions.
• State estimator/filter: $s_t = \mathrm{Enc}_\phi(s_{t-1}, x_t, a_{t-1})$ or $P_\phi(s_t | x_{1:t}, a_{1:t-1})$ when inferring state from raw observations.

State representations can be explicit (e.g., 3D geometry, semantic maps, physical parameters, object-centric slots) or implicit (context windows, compressed memories, neural hidden state). Hybrid forms are also emerging, e.g., composite structures concatenating global geometric features, appearance embeddings, and multi-modal signals (Wang et al., 22 Jan 2026).

2. State Construction Paradigms

The domain divides state construction into two primary classes (Wang et al., 22 Jan 2026):

• Implicit State — Context Management
  • Sliding window attention: the state is embedded in the last $k$ frames processed by a Transformer or diffusion model.
  • Memory mechanisms: approaches such as compressed key/value buffers, recurrent hidden state, retrieval-augmented attention, or block-scanned state-spaces (Po et al., 26 May 2025, Oshima et al., 2 Dec 2025).
  • While this ensures flexibility and maximal visual fidelity, memory is bounded and persistence is “window-limited,” risking forgotten objects/layouts beyond the context span.
• Explicit State — Latent Compression and Geometric Modeling
  • Encoders compress clips or histories into a latent state vector or 3D/4D field (Hu et al., 5 Jun 2025, Zheng et al., 8 Jan 2026).
  • Linear or nonlinear state-space models propagate state with constant memory cost; priors such as SSMs, Gaussian fields, or graphical models may be used to structure the dynamics (Po et al., 26 May 2025, Lu et al., 2023).
  • Explicit object- or geometry-centric states—such as segmented 3D Gaussian fields (Hu et al., 5 Jun 2025), 3D/4D occupancy grids (Lu et al., 2023), or background-point clouds plus per-object Gaussian trajectories (Zheng et al., 8 Jan 2026)—enable direct physical manipulation, compositional reasoning, and accurate simulation under occlusion.

Table: Notable Explicit State Representations

Model	State Variable	Structure
DSG-World (Hu et al., 5 Jun 2025)	Dual 3D Gaussian fields	$(x, \Sigma, \alpha, c, s)$ per primitive
VerseCrafter (Zheng et al., 8 Jan 2026)	$(B, \{O_o(t)\})$	Point cloud + time-varying 3D Gaussians
WoVoGen (Lu et al., 2023)	$W_t \in \mathbb{R}^{X \times Y \times Z \times C}$	World volume (semantic & map channels)
WorldPack (Oshima et al., 2 Dec 2025)	Memory-packed VAE latents	Hierarchical, trajectory-packed tokens

3. Dynamics Modeling, Control, and Action Conditioning

State-aware models enforce causality and enable interaction by modifying the transition kernel $f$ with action or control inputs. Key strategies include:

• Autoregressive masking: transformers/denoisers are restricted to attend only to past or present context to prevent information leakage from future frames (Wang et al., 22 Jan 2026).
• Forcing strategies: Diffusion Forcing (Chen et al., 28 May 2025, Po et al., 26 May 2025) and self-forcing schemes inject noise into context to teach robust denoising and mitigate compounding error through rollouts.
• Action/State Modulation: Adaptive Layer Normalization (AdaLN) applies per-timestep scale/shift to intermediate activations based on action vector and global state embeddings (Chen et al., 28 May 2025).
• 4D Geometric Control: VerseCrafter (Zheng et al., 8 Jan 2026) enables direct specification of object and camera trajectories as time-varying 3D Gaussian distributions, rendered as control maps and fused into video generator backbones via lightweight adapters, supporting accurate, physics-informed generation and scene manipulation.

In addition, models such as WorldPack (Oshima et al., 2 Dec 2025) and DeepVerse (Chen et al., 1 Jun 2025) integrate geometry-aware memory retrieval, ensuring that representations remain consistent under significant viewpoint changes and long-horizon rollouts.

4. Memory Systems and Long-term Consistency

A persistent challenge is maintaining state fidelity over long horizons:

• Compressed trajectory packing: Hierarchical compression of past latent tokens (WorldPack (Oshima et al., 2 Dec 2025)) allows O(10–20$\times$) more history to influence current predictions without quadratic computation.
• Block-wise causal SSMs: Long-context SSMs break the spatial grid into blocks, updating compact per-block state across arbitrarily long sequences, combining short-range spatial attention and O(1) per-frame memory (Po et al., 26 May 2025).
• External retrieval: VRAG (Chen et al., 28 May 2025) introduces a buffer of past frames with explicit global state (position/orientation), and a retrieval scheme using geometric similarity, ensuring the model can recall distant but relevant memory for robust denoising and anchor world coherence.
• Aggregated spatial memory: Persistent embodied world models (Zhou et al., 5 May 2025) construct a live 3D voxel map, updated by back-projecting features from each newly generated video segment, and use this spatial memory as input for subsequent video predictions.

Theoretical analysis (Wang et al., 22 Jan 2026) and experimental evidence confirm that explicit or compressed memory is essential to counteract information loss, object drift, and topological inconsistency—problems endemic to naive autoregressive or fixed-context video generators.

5. Bidirectional Consistency, Supervision, and Training Objectives

State-aware models employ a range of objectives to enforce physical and semantic alignment:

• Reconstruction losses: Standard losses over color, depth, and segmentation features (Hu et al., 5 Jun 2025).
• Bidirectional alignment: Training with object-level cross-state supervision (transforming between paired observed states) ensures that scene representations remain mutually consistent under known object reconfigurations (Hu et al., 5 Jun 2025).
• Pseudo-intermediate states: Symmetric alignment through geometrically interpolated pseudo-states forces correspondence in ambiguous regions (occlusions, mutual visibility), further tying representations across states (Hu et al., 5 Jun 2025).
• Collaborative co-pruning: Nearest-neighbor geometric consistency checks prune unmatched primitives to eliminate artifacts and enforce mutual explainability (Hu et al., 5 Jun 2025).
• Functional regularizers and auxiliary losses: Persistence penalties, causal Jacobian constraints, and contrastive causal losses can be used to promote physical plausibility (Wang et al., 22 Jan 2026).
• Denoising and flow-matching: Diffusion or flow-matching losses (on images, latents, control trajectories) are standard in DiT and related architectures (Chen et al., 28 May 2025, Hu et al., 25 Dec 2025).

6. Applications: Manipulation, Simulation, and Planning

State-aware video world models enable expanded capabilities beyond passive video forecasting:

• Novel-view and state synthesis: Explicit 3D/4D state allows synthesis of arbitrary new viewpoints, dynamic reconfiguration of objects, and scene manipulation (e.g., Gaussian co-pasting in DSG-World (Hu et al., 5 Jun 2025); direct control editing in VerseCrafter (Zheng et al., 8 Jan 2026)).
• Interactive video generation: Action-conditioned video rollout and interactive planning via autoregressive or diffusion models (Chen et al., 28 May 2025, Hu et al., 25 Dec 2025, Huang et al., 8 Dec 2025).
• Sim-to-real transfer: Explicit geometric modeling or persistent memory supports reliable transfer for real-to-simulation and simulation-to-real tasks, as shown in robotics and navigation benchmarks (Hu et al., 5 Jun 2025, Zhou et al., 5 May 2025, Hu et al., 25 Dec 2025).
• Closed-loop reinforcement learning: State-aware world models can serve as efficient, high-fidelity simulators for training and improving RL policies, with a closed-loop refinement structure improving both model and policy iteratively (Liu et al., 6 Feb 2026).
• Generalization and zero-shot adaptation: Multimodal and explicit state representations (UnityVideo (Huang et al., 8 Dec 2025), AstraNav-World (Hu et al., 25 Dec 2025)) show strong generalization across domains and scenarios, with transfer to real-world or previously unseen test conditions.

7. Benchmarks, Evaluation, and Open Challenges

Functional evaluation shifts from pure visual metrics (FID, LPIPS, PSNR, SSIM) to task-driven and causal assessments (Wang et al., 22 Jan 2026):

• Persistence and consistency: Metrics track error over long-term rollouts (e.g., spatial retrieval, reasoning, and revisit consistency in WorldPack (Oshima et al., 2 Dec 2025) and Long-Context SSM (Po et al., 26 May 2025)).
• Causality and controllability: Intervention-based rollouts and action-fidelity scores quantify the model’s capacity to mediate cause–effect under varying controls (Wang et al., 22 Jan 2026).
• Downstream performance: Absolute trajectory error (ATE), planning success rates, object manipulation accuracy, and navigation scores serve as end-to-end validators for the internal world model (Zhou et al., 5 May 2025, Hu et al., 25 Dec 2025, Liu et al., 6 Feb 2026).
• Modal and semantic segmentation: Multimodal benchmarks (segmentation, depth, pose) confirm state reasoning in models like UnityVideo (Huang et al., 8 Dec 2025).

Current open challenges include integrating independently dynamic objects with static 3D state (Zhou et al., 5 May 2025), scaling to unbounded environments, combining more diverse physical modalities, and learning memory/retrieval policies end-to-end for truly persistent, general-purpose world simulation (Wang et al., 22 Jan 2026, Huang et al., 8 Dec 2025).

---

In summary, the state-aware video world model paradigm is characterized by the explicit construction, propagation, and utilization of structured world state throughout the video generation process, whether through geometric primitives, volumetric fields, latent compressions, or hybrid memory. This technical trajectory is foundational for progress in embodied AI, interactive simulation, action-conditioned video generation, and the broader goal of robust, general-purpose world modeling. Key references: (Hu et al., 5 Jun 2025, Chen et al., 28 May 2025, Wang et al., 22 Jan 2026, Po et al., 26 May 2025, Oshima et al., 2 Dec 2025, Huang et al., 8 Dec 2025, Lu et al., 2023, Zhou et al., 5 May 2025, Zheng et al., 8 Jan 2026, Liu et al., 6 Feb 2026, Chen et al., 1 Jun 2025, Xiang et al., 2024, Hu et al., 25 Dec 2025).