Interactive Video Generation Frameworks

• Interactive Video Generation Frameworks are systems that combine generative modeling, interactive control, memory architectures, and dynamic simulation for real-time video synthesis.
• These frameworks integrate conditioning, generation, and control mapping modules to enable high-fidelity visuals and user-directed narrative alterations.
• Applications span gaming, digital avatars, and live streaming, leveraging diffusion and transformer-based models to achieve low-latency, coherent video outputs.

Interactive video generation frameworks represent a convergence of generative modeling, real-time user control, memory architecture, and dynamic world simulation. These systems have rapidly advanced beyond passive text- or image-to-video synthesis to support fine-grained, interactive behaviors—enabling users to direct narratives, control agents and cameras, and shape virtual environments with low latency and high visual fidelity. The field spans architectures based on diffusion, autoregressive generation, hybrid transformers, and plug-and-play modules, and serves domains including gaming, entertainment, simulation, digital avatars, and real-time streaming.

1. Core Architectural Components

Modern interactive video generation frameworks typically comprise modules for conditioning, generation, memory, control mapping, and system-level optimizations. For example, systems such as RealCam-I2V (Li et al., 14 Feb 2025), InteractiveVideo (Zhang et al., 2024), and GameGen-X (Che et al., 2024) all leverage foundational backbone generative models—usually diffusion-based (e.g., DiT, MM-DiT)—augmented by explicit architectural innovations for interaction.

Key architectural modules seen across leading frameworks:

• Condition encoders: Multimodal inputs (text, image, audio, pose, trajectory drags) are projected into control tokens, embeddings, or affine modulation parameters. RealCam-I2V applies monocular metric depth estimation and 3D reconstruction to support camera-trajectory conditioning, while InteractiveVideo supports painting, text, and drag instructions via residual fusion directly into the denoising step (Li et al., 14 Feb 2025, Zhang et al., 2024).
• Generative core: Typically a temporal or spatiotemporal diffusion model in the latent space, possibly with transformer-based blocks (e.g., DiT, MMDiT, MSDiT). Architectures such as TV2TV (Han et al., 4 Dec 2025) interleave text and video using a Mixture-of-Transformers, while FaustFlow-R1 (Wang et al., 15 Jan 2026) and MotionStream (Shin et al., 3 Nov 2025) adapt DiT architectures for real-time chunkwise streaming and causal rollout.
• Memory mechanisms: Sliding-window or explicit memory modules (FramePack in Yume (Mao et al., 23 Jul 2025), long/short-term memory queues in FlowAct-R1) maintain coherence, prevent drift, and support infinite or extended video generation.
• Control mapping: User signals are mapped via learned or engineered branches (e.g., camera-path editors in RealCam-I2V, drag-FiLM and cross-attention in Puppet-Master (Li et al., 2024), keyboard/mouse to continuous trajectories in Hunyuan-GameCraft (Li et al., 20 Jun 2025), quantized action vocabularies in Yume).
• Decoding and system optimization: Latent sequences are decoded through VAE heads, sometimes coupled with adversarial or flow-matching losses. Model quantization, pipeline parallelism, and attention masks (e.g., embedding reuse, frame sinks) support real-time operation (Yu et al., 6 Jun 2025, Feng et al., 10 Nov 2025, Yang et al., 26 Sep 2025).

2. Control Modalities and Interaction Mechanisms

Interactive video generation transcends passive prompts by allowing dynamic, fine-grained, and sometimes multimodal user intervention during synthesis:

• Direct trajectory or object control: RealCam-I2V supports user-drawn camera trajectories with SE(3) parameterization, while Puppet-Master enables drag-and-drop part-level controls for motion of semantic segments (Li et al., 14 Feb 2025, Li et al., 2024).
• Image and region manipulation: InteractiveVideo allows users to paint or semantically edit a reference key-frame, with each edit dynamically injected as a residual into diffusion denoising, enabling iterative WYSIWYG (What-You-See-Is-What-You-Get) control (Zhang et al., 2024).
• Action and keyboard mapping: Quantized camera/action tokens from keyboard or mouse are fused as control signals (Yume, Hunyuan-GameCraft, GameGen-X), supporting smooth agent navigation and scene manipulation (Mao et al., 23 Jul 2025, Li et al., 20 Jun 2025, Che et al., 2024).
• Textual interventions: TV2TV, GameGen-X, and MIDAS mediate narrative or behavioral changes through injected text, which can be appended or modified at any generation step without disrupting previous semantic context (Han et al., 4 Dec 2025, Che et al., 2024, Chen et al., 26 Aug 2025).
• Multimodal and streaming prompts: MIDAS and LLIA integrate audio, pose, and text in real time for interactive digital humans and avatars. StreamDiffusionV2 and MotionStream support live prompt switching and temporal sliding via KV reuse and sink tokens (Chen et al., 26 Aug 2025, Yu et al., 6 Jun 2025, Shin et al., 3 Nov 2025, Feng et al., 10 Nov 2025).

Table: Example Modalities Supported by Leading Frameworks

Framework	Camera/Traj.	Textual	Audio	Object/Region	Keyboard/Mouse	Painting/Drag
RealCam-I2V	✓	✓	–	–	–	–
InteractiveVideo	–	✓	–	✓	–	✓
Hunyuan-GameCraft	✓	✓	–	–	✓	–
Puppet-Master	–	–	–	✓	–	✓
TV2TV	–	✓	–	–	–	–
LLIA	–	–	✓	–	–	–
MIDAS	–	✓	✓	–	–	–
Yume	✓	✓	–	–	✓	–

3. Memory, Consistency, and Error Accumulation

A major technical challenge is maintaining temporal and spatial consistency across long sequences, especially under user-driven interactive control. Models must avoid drift, preserve scene geometry and identity, and ensure action-response fidelity.

• Sliding window and KV caches: Rolling attention windows with periodically refreshed frame sinks or anchor tokens (see LongLive (Yang et al., 26 Sep 2025), StreamDiffusionV2 (Feng et al., 10 Nov 2025), MotionStream (Shin et al., 3 Nov 2025)) limit error accumulation and support real-time inference at constant computational cost.
• Scene-constrained and motion-aware shaping: RealCam-I2V introduces scene-constrained noise shaping, conditioning high-noise stages to project samples back onto metric 3D previews, which constrains the generation to the user-defined camera path (Li et al., 14 Feb 2025). StreamDiffusionV2 further combines motion-aware noise scheduling to adapt denoising strength according to detected scene dynamics (Feng et al., 10 Nov 2025).
• Dynamic frame eviction: StableWorld applies ORB-based geometric comparison across temporal anchors in a sliding window, evicting the most drifted historic frames to prevent scene collapse over long rollouts. This approach is architecture-agnostic and shown to be effective across diverse generation backbones (Yang et al., 21 Jan 2026).
• Retrieval-augmented memory and global state: Learning World Models for Interactive Video Generation (Chen et al., 28 May 2025) introduces VRAG, which retrieves latent historical frames by global state similarity (e.g., position, camera yaw), explicitly reinserting geometrically and semantically relevant history to reduce compounding prediction error.

4. Training Paradigms, Losses, and System-Level Adaptations

Training interactive frameworks combines classic denoising objectives with control- and memory-specific losses:

• Standard diffusion MSE loss: Denoising score-matching remains fundamental. Action, camera pose, and content conditioning is typically injected into the denoising U-Net, sometimes via AdaLN or cross-attention (Li et al., 20 Jun 2025, Chen et al., 28 May 2025, Che et al., 2024).
• Control and trajectory regularization: Camera or action consistency losses, e.g., as in RealCam-I2V, penalize deviation between generated and conditioned poses through explicit alignment of predicted and reference SE(3) transformations (Li et al., 14 Feb 2025).
• Memory and history matching: Explicit losses on history preservation or global state ensure that retrieval- or memory-augmented modules remain faithful to the intended past states (Chen et al., 28 May 2025, Yang et al., 26 Sep 2025).
• Hybrid and adversarial distillation: Model acceleration in frameworks like Yume involves adversarial distillation, reducing diffusion steps from 50 to 14 with negligible quality loss. LLIA applies a latent consistency objective augmented by adversarial losses for robust, expressive face synthesis at high frame rates (Mao et al., 23 Jul 2025, Yu et al., 6 Jun 2025).
• Chunkwise, streaming, and self-forcing adaptation: Real-time deployment mandates conversion of bidirectional diffusion to AR or chunkwise AR rollout (see FlowAct-R1 (Wang et al., 15 Jan 2026), LongLive (Yang et al., 26 Sep 2025), MotionStream (Shin et al., 3 Nov 2025)), with self-forcing and distribution-matching distillation to close the train-test and offline-online gaps.

5. Application Domains and Benchmark Results

Interactive video frameworks underpin applications in game engines, virtual agents, open-world simulation, digital humans, and live streaming.

• Game engines and world simulators: IGV-based models (GameGen-X (Che et al., 2024), Hunyuan-GameCraft (Li et al., 20 Jun 2025), Yume (Mao et al., 23 Jul 2025)) demonstrate autoregressive, key/mouse, and text-controllable multi-minute generation, with high action controllability and dynamic scene consistency verified on VBench, Yume-Bench, and custom open-world video benchmarks.
• Dynamic avatars and talking heads: LLIA and MIDAS achieve streaming, low-latency, high-FPS synthesis of audio-driven expressive avatars, supporting nuanced conversational state control and multimodal conditioning (Yu et al., 6 Jun 2025, Chen et al., 26 Aug 2025).
• Part-level and object-centric control: Puppet-Master allows drag-controlled part- and object-level motion synthesis, evaluated with PSNR, SSIM, LPIPS, FVD, and flow-error metrics, outperforming prior drag-based generators on Drag-a-Move and Human3.6M (Li et al., 2024).
• Real-time live streaming: StreamDiffusionV2 delivers 60+ FPS, <0.5 s TTFF, and per-frame SLO guarantees for multi-GPU, scalable live generation, using batching, KV pipeline orchestration, and noise scheduling (Feng et al., 10 Nov 2025).
• Editing and storyboarding: TV2TV enables open-ended video-text interleaving with mid-sequence text edits, supporting storytelling and prompt-directed gameplay with high prompt alignment and user preference scores (Han et al., 4 Dec 2025).

Selected Evaluation Metrics and Results:

Model	FPS	FVD	Action Metric	Notable Annotation
MotionStream	29.5	~LPIPS=0.44	PSNR≈16.2	2 orders mag. faster than prior methods (Shin et al., 3 Nov 2025)
RealCam-I2V	N/A	–13% over CamI2V	–27% TransErr	Absolute-scale camera control, RealEstate10K (Li et al., 14 Feb 2025)
Hunyuan-GameCraft	6.6 (distilled)	1554.2	RPE_trans=0.08	1M+ gameplay dataset, OOD generalization (Li et al., 20 Jun 2025)
TV2TV	N/A	User study	Intervention accuracy	Text-video interleaving, story editing (Han et al., 4 Dec 2025)
Puppet-Master	N/A	FVD=247	Flow-error=12.2/3.5	Part-level drag, Objaverse-HQ, zero-shot (Li et al., 2024)

6. Limitations, Challenges, and Future Research

Despite rapid progress, several open challenges persist:

• Long-horizon and memory management: Error drift and scene instability in infinite or extended rollouts still arise, especially in static or low-dynamic scenes. Solutions such as StableWorld’s frame eviction and explicit global state memory are effective but introduce new hyperparameters and potential for over-eviction in low-texture content (Yang et al., 21 Jan 2026, Chen et al., 28 May 2025).
• Real-time guarantees: While systems like StreamDiffusionV2 and LongLive have reached real-time operation with hundreds of frames and prompt switching, model compression, quantization artifacts, and hardware variability remain practical bottlenecks (Feng et al., 10 Nov 2025, Yang et al., 26 Sep 2025).
• Open-domain and fine-grained control: Generalization to complex, OOD domains (e.g., multi-object manipulation, advanced physics, concept editing) is still in early stages. The integration of language, symbolic reasoning, multimodal LLMs, and causal/physics-aware modules is an active area (Han et al., 4 Dec 2025, Che et al., 2024, Yu et al., 30 Apr 2025).
• User interaction complexity: Multimodal and iterative editing (e.g., InteractiveVideo) introduces pipeline latency, modality fusion challenges, and post-hoc temporal smoothing demands (Zhang et al., 2024).
• Benchmarking and reproducibility: VBench, Yume-Bench, and AnimateBench provide initial standardized metrics for visual quality, controllability, and dynamics, but comprehensive, open-domain, multimodal benchmarks and physics-causality metrics are needed (Mao et al., 23 Jul 2025, Zhang et al., 2024, Yu et al., 30 Apr 2025).

7. Theoretical and System-Level Foundations

The evolution of interactive video generation frameworks illustrates cross-pollination from generative modeling, control theory, computer graphics, and systems optimization. IGV, as defined in seminal surveys and position papers (Yu et al., 30 Apr 2025, Yu et al., 21 Mar 2025), is decomposed into five synergistic modules—generation, control, memory, dynamics, and intelligence. This modular perspective enables rapid innovation and composability, guiding the field toward the realization of generative game engines, open-world simulators, digital human communication, and self-evolving media ecosystems.

The integration of explicit control, strong memory, physical principles, and causal reasoning forms the blueprint for next-generation frameworks, aligning technical objectives (e.g., latency, coherence, controllability, realism) with emerging application domains ranging from autonomous systems to live interactive entertainment.