Latent Video Diffusion Models (LVDMs)

Updated 8 February 2026

Latent Video Diffusion Models are generative frameworks that compress video clips into spatiotemporal latent spaces via VAEs and refine them using conditional diffusion processes.
They employ a three-stage pipeline—encoding, latent diffusion with U-Net/Transformer backbones, and decoding—to achieve efficient and high-quality video reconstruction.
Innovations in spectral regularization, hierarchical structures, and corruption-aware training enhance temporal coherence, scalability, and practical performance in extended video generation.

Latent Video Diffusion Models (LVDMs) are a class of generative video frameworks that leverage the representational benefits of deep latent spaces—structured by Variational Autoencoders (VAEs)—with the temporal/structural modeling power of conditional diffusion processes. LVDMs operate by first compressing high-dimensional video clips into a spatiotemporal latent space, then learning a diffusion-based denoising backbone in this manifold. This enables photorealistic, long-range generative modeling at reduced compute cost and enhanced scalability relative to pixel-space alternatives.

1. Core Architecture of Latent Video Diffusion Models

LVDMs are typically constructed as a three-stage pipeline:

Encoding: An input video $v \in \mathbb{R}^{F \times H \times W \times 3}$ is mapped to a lower-dimensional latent $z_0$ via a pretrained video VAE encoder $E_v$ . Typical compression ratios are on the order of 1:32 to 1:192, achieved by joint spatial and temporal downsampling using 3D convolutions, wavelet transforms, or patchification (He et al., 2022, HaCohen et al., 2024, Li et al., 2024, Cheng et al., 18 Mar 2025).
Latent Diffusion Process: A Markovian forward process $q(z_t \mid z_0)$ introduces noise to the latent representation over $T$ steps:

$q(z_t \mid z_0) = \mathcal{N}\left(z_t; \sqrt{\bar \alpha_t} z_0, (1 - \bar \alpha_t) I\right)$

A conditional denoiser (U-Net or Transformer) parameterized by $\theta$ predicts the noise in the reverse process $p_\theta(z_{t-1}|z_t, c)$ , where $c$ encodes side information (e.g., text, image) (He et al., 2022, Blattmann et al., 2023, HaCohen et al., 2024).

Decoding: The final denoised latent $z_0$ is mapped back to pixel space via $D_v$ to reconstruct the video $\hat v$ (Wu et al., 2024, Chen et al., 2024).

This factorized representation allows for significant reductions in computational resources, robust learning of temporal dependencies, and direct adaptation of image-diffusion advancements into the video domain (He et al., 2022, Blattmann et al., 2023).

2. Advances in Video VAE Compression and Latent Structure

The efficiency, fidelity, and scalability of LVDMs depend critically on the properties of the underlying video VAE.

Omni-Dimensional Compression: The "OD-VAE" introduces 3D causal convolutional stacks, achieving $8\times$ spatial and $4\times$ temporal reduction compared to vanilla 2D VAEs, while maintaining high PSNR/SSIM and halving GPU memory usage (Chen et al., 2024).
Wavelet and Patch-based Designs: WF-VAE leverages a multi-level 3D Haar wavelet transform for frequency decomposition, enabling high-throughput, low-memory latent extraction and seamless block-wise inference via causal convs and caching (Li et al., 2024). LeanVAE combines non-overlapping patchification, wavelet transforms, and compressed sensing, attaining $44\times$ faster encoding with comparable or better LPIPS/rFVD as previous VAEs (Cheng et al., 18 Mar 2025).
Keyframe and Temporal Decomposition: IV-VAE proposes a keyframe-based temporal compression (KTC) coupled with group causal convolutions (GCConv), splitting the latent into spatial and temporal branches, with only half of channels learning new temporal priors—yielding reduced flicker and improved FVD/SOTA performance on high-res benchmarks (Wu et al., 2024).
Spectral Regularization: SSVAE introduces Local Correlation Regularization (LCR) and Latent Masked Reconstruction (LMR) to bias the latent space toward low frequencies and "few-mode" channel dominance, leading to $3\times$ faster diffusion training and $10\%$ uplift in video reward (Liu et al., 5 Dec 2025).

A summary of empirical throughput, memory, and quality results for state-of-the-art VAEs is shown below (metrics on 33 frames @256×256) (Li et al., 2024):

Method	PSNR ↑	LPIPS ↓	Memory (MB)	Throughput (s)
OD-VAE	30.69	0.0553	31944	0.0945
WF-VAE-L(16ch)	35.76	0.0230	9000	0.0600
LeanVAE	26.04	0.0899	—	0.46 (@768p)

Advances in content-aware multi-level temporal compression (MTC-VAE) further enable variable-rate latent representations; by dynamically adjusting compression rates per video segment, one can nearly double speed and halve memory while preserving VBench generation scores (Dong et al., 1 Feb 2026).

3. Diffusion Process: Denoising Architectures and Temporal Modeling

The diffusion backbone is typically a U-Net (inherited or video-specific) or Transformer that models both spatial and temporal dependencies in the latent space.

U-Net Extensions: Spatial U-Nets from pretrained image models are temporally "inflated" by inserting 3D convolutional, temporal self-attention, or causal attention blocks (Blattmann et al., 2023, Chen et al., 2024, Gu et al., 2023). Hybrid strategies—alternating 2D/3D blocks—trade off speed and quality (Chen et al., 2024).
Transformer Backbones: Recent models such as LTX-Video utilize a pure spatiotemporal self-attention transformer, running over highly compressed latent grids (32x32x8), with cross-attention for conditioning (HaCohen et al., 2024).
Temporal Consistency Control: Additional mechanisms (e.g., spectral-structured noise, group causal convs, tail initialization, temporal–spatial attention, cross-attention fusion, or temporal-tiling) are introduced to ensure frame-to-frame coherence and facilitate efficient long-sequence inference (Liu et al., 2024, Wu et al., 2024, HaCohen et al., 2024).

Sampling in the latent domain enables orders-of-magnitude reductions in compute—typical models operate with $32\times$ – $192\times$ fewer tokens than pixel-space (HaCohen et al., 2024), with throughput scaling nearly linearly as per-frame activations shrink.

4. Hierarchical and Long-Range Video Generation

To address temporal drift and error accumulation in long videos, LVDMs often employ hierarchical or autoregressive structures:

Hierarchical Diffusion: As described in "Latent Video Diffusion Models for High-Fidelity Long Video Generation" (He et al., 2022), key frames are predicted at intervals via a sparse model, with intermediate frames interpolated by a dedicated model. This allows for generation of >1,000 frames with controlled quality decay.
Conditional Latent Perturbation: When generating conditioned blocks, explicit noise is added to the conditioning keys, forcing the model to remain robust to errors and reducing accumulated drift.
Prediction and Interpolation: Fine-tuning only temporal blocks enables pretrained spatial models to be reused; interpolation models are trained to infill missing or slow-motion frames, crucial for video upsampling or slow-motion effects (Blattmann et al., 2023).

These innovations enable generation of minutes-long, temporally coherent clips at 256×256 or higher resolutions (He et al., 2022, Blattmann et al., 2023).

5. Practical Trade-offs, Training Techniques, and Empirical Findings

LVDMs support numerous architectural and procedural trade-offs:

Speed/Quality Variants: OD-VAE formalizes four variants trading 3D/2D conv blocks, showing that encoder–outer-2D (decoder all-3D) yields $2\times$ diffusion speedup without sacrificing FVD (Chen et al., 2024).
Initialization: Tail-initialization of 3D convs with pretrained 2D kernels ensures compatibility with legacy models and stabilizes early VAE convergence (Chen et al., 2024).
Block-wise and Tiled Inference: Temporal tiling with overlapping groups and causal cache mechanisms allow arbitrary-length generation with consistent boundaries, leveraging the translation-invariance of temporal convs (Li et al., 2024, Chen et al., 2024).
Spectral and Content Biasing: LCR and LMR regularizers guide latent statistics for enhanced diffusability; best practices include tuning local correlation thresholds and mask rates for dataset-specific structure (Liu et al., 5 Dec 2025).

Empirical results demonstrate that state-of-the-art LVDMs (e.g., using WF-VAE, OD-VAE, IV-VAE, SSVAE) surpass pixel-space approaches and vanilla video VAEs in FVD, PSNR, IS, CLIPSim, training speed, efficiency, and visual appeal across domains such as WebVid-10M, UCF101, SkyTimelapse, and MovieGenBench (Chen et al., 2024, Li et al., 2024, Wu et al., 2024, Liu et al., 5 Dec 2025, Cheng et al., 18 Mar 2025).

6. Robustness, Extensions, and Practical Applications

Recent research addresses robustness to conditioning noise and alternative application domains:

Corruption-Aware Training (CAT-LVDM): By injecting structured, batch-centered semantic noise (BCNI) or spectrum-aware contextual noise (SACN) into the conditioning embeddings, CAT-LVDM achieves state-of-the-art FVD gains ( $-31.9\%$ on WebVid-2M/MSR-VTT/MSVD; $-12.3\%$ on UCF101) and improves smoothness/generalization under caption noise, with theoretical bounds guaranteeing improved mixing and reduced score drift (Maduabuchi et al., 24 May 2025).
Watermarking and Ownership: LVMark establishes robust video watermarking in LVDMs by embedding signals in low-frequency latent decoder weights, leveraging 3D-DWT for resilient, temporally consistent, and invisible watermarks recoverable even after aggressive postprocessing (Jang et al., 2024).
Video Editing and Multi-Source Fusion: Models such as FLDM perform fusion of independent text-to-image and text-to-video latents during the denoising process, balancing textual fidelity and temporal consistency, with competitive CLIP-based metrics versus baseline frame-wise/inpainting approaches (Lu et al., 2023).
Local Editing and Masking: Latent attention mechanisms and autonomous mask manufacture (via cross-attention statistics) enable targeted editing in real video, outperforming both frame-wise and cascade-fusion baselines in real-world scenarios (Liu et al., 2024).

LVDMs thus support high-fidelity video generation, robust editing, and practical watermarking at scale, leveraging an expanding array of latent regularization, architectural, and inference advancements.

7. Limitations and Future Directions

Outstanding challenges include:

Ultra-long Video and High-Resolution Scaling: While current LVDMs handle up to several thousand frames and megapixel resolutions, error accumulation and compute requirements remain bottlenecks for 4K or 10,000+ frame generation. Adaptive hierarchical, recurrent, or flow-aware extensions are under investigation (He et al., 2022, Blattmann et al., 2023).
Latent Structure Learning: Overly aggressive spectral biasing or compression can degrade motion fidelity; further work is needed on dataset-adaptive latent shaping, possibly with learned transforms or discrete/continuous hybrid latents (Liu et al., 5 Dec 2025, Cheng et al., 18 Mar 2025).
Conditional Alignment and Multi-Modal Fusion: Robust alignment of multi-modal inputs, especially under semantic noise or misaligned dataset captions, remains nontrivial despite advances in corruption-aware training (Maduabuchi et al., 24 May 2025).
Transferability: While modular temporal/spatial block strategies allow for limited parameter transplantation across backbone models, full cross-modal or multi-domain generalization is only partially demonstrated (Blattmann et al., 2023).

Potential research trajectories involve learned flow/optical-motion integration, memory-efficient transformer scaling, adaptive sampling strategies, and plug-and-play personalization/fine-tuning frameworks.

References:

"OD-VAE: An Omni-dimensional Video Compressor for Improving Latent Video Diffusion Model" (Chen et al., 2024)
"WF-VAE: Enhancing Video VAE by Wavelet-Driven Energy Flow for Latent Video Diffusion Model" (Li et al., 2024)
"LeanVAE: An Ultra-Efficient Reconstruction VAE for Video Diffusion Models" (Cheng et al., 18 Mar 2025)
"Improved Video VAE for Latent Video Diffusion Model" (Wu et al., 2024)
"Delving into Latent Spectral Biasing of Video VAEs for Superior Diffusability" (Liu et al., 5 Dec 2025)
"Latent Video Diffusion Models for High-Fidelity Long Video Generation" (He et al., 2022)
"Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models" (Blattmann et al., 2023)
"LTX-Video: Realtime Video Latent Diffusion" (HaCohen et al., 2024)
"Corruption-Aware Training of Latent Video Diffusion Models for Robust Text-to-Video Generation" (Maduabuchi et al., 24 May 2025)
"LVMark: Robust Watermark for Latent Video Diffusion Models" (Jang et al., 2024)
"Fuse Your Latents: Video Editing with Multi-source Latent Diffusion Models" (Lu et al., 2023)
"Blended Latent Diffusion under Attention Control for Real-World Video Editing" (Liu et al., 2024)
"MTC-VAE: Multi-Level Temporal Compression with Content Awareness" (Dong et al., 1 Feb 2026)