Masked Video Transformers

• Masked video transformers are deep architectures that extend traditional transformers to video by masking spatio-temporal tokens for robust representation learning.
• They employ varied strategies like uniform, tube, and token compression masking to optimize self-supervised pretraining, compression, and generative tasks.
• Empirical studies reveal that high masking ratios exploit temporal redundancy, leading to efficient video understanding and improved performance in tasks such as inpainting and prediction.

Masked video transformers are transformer-based architectures that learn and process video data through masking strategies designed to induce robust spatio-temporal representations, improve generative modeling, facilitate video compression, and enable tasks such as inpainting, prediction, super-resolution, editing, and loss-resilient transmission. These models build on masked language modeling and masked image modeling paradigms, extending them to the temporal and spatial domains of video by predicting masked (hidden or dropped) tokens, patches, or tubelets from observed context. The masking can be structured to configure self-supervised objectives, enable efficient training or inference, facilitate entropy/multiple description coding, or impose architectural sparsity for computational efficiency.

1. Architectural Foundations and Principal Variants

Masked video transformers extend the transformer paradigm popularized in language and vision domains to video by introducing masking at the level of video tokens—often corresponding to spatio-temporal patches (tubelets), latent codes, or quantized features. Architectures typically follow one of:

• Vanilla Vision Transformers (ViTs): Used with uniform (patch or tube) masking, e.g., VideoMAE (Tong et al., 2022), OmniMAE (Girdhar et al., 2022), Data Collection-free Masked Video Modeling (Ishikawa et al., 2024), and Recurrent Video Masked Autoencoders (Zoran et al., 15 Dec 2025).
• Hierarchical Video Transformers: E.g., Video Swin Transformer backbone with masking in BEVT (Wang et al., 2021) and VIOLET (Fu et al., 2021, Fu et al., 2022).
• Bidirectional Masked Transformers: Employed for generative masked token prediction and entropy modeling, as in NeuralMDC (Hu et al., 2024), where past and current tokens are jointly attended.
• Diffusion Transformers: General-purpose video generation and completion by integrating temporal and spatial attention with masking into diffusion models (VDT (Lu et al., 2023)), and dual masking for scene alignment and extension (Mask$^2$DiT (Qi et al., 25 Mar 2025)).
• Masked Transformers for Inpainting and Editing: Models such as Deficiency-Aware Masked Transformer (DMT) (Yu et al., 2023) and MaskINT (Ma et al., 2023) utilize masking to inpaint or interpolate video content.
• Motion/Efficiency-Oriented Models: Masked Appearance-Motion Modeling (MAM²) (Song et al., 2022), Motion-Guided Token Compression (Feng et al., 2024), and MIA-VSR (Zhou et al., 2024) exploit temporal redundancy or similarity to guide masking and computational reduction.

Common to these designs is a separation between encoder and decoder (or predictor) modules, with masking applied to input features or latent representations before specific transformer blocks. In certain generative or compression settings, auxiliary modules (tokenizers, VAEs, entropy models, or codebooks) are used.

2. Masking Strategies and Objectives

The strategies for masking in masked video transformers are tailored to dataset structure, intended application, and computational constraints:

• Uniform Random Masking: Applied i.i.d., typically with high ratio (95% for videos in OmniMAE (Girdhar et al., 2022); 90–95% in VideoMAE (Tong et al., 2022)), challenging the model to leverage all available spatio-temporal context.
• Tube/Block Masking: 2D blocks replicated along the temporal axis (tube masking, as in VideoMAE, BEVT, MAM²) prevent trivial copying and enforce reliance on longer temporal context (Tong et al., 2022, Song et al., 2022, Wang et al., 2021).
• Spatial-Temporal Block Masking: Unified randomly sampled spatio-temporal mask for completion/generation (VDT (Lu et al., 2023), DMT (Yu et al., 2023)).
• Token Compression Masking: Velocity-based or variance-based masks that exclude tokens with low motion or redundancy (“motion guided" masking in MGTC (Feng et al., 2024); adaptive block-wise similarity masking in MIA-VSR (Zhou et al., 2024)).
• Data-Driven and Attentional Masking: Masking based on transformer attention scores or domain importance (attended masking in VIOLET (Fu et al., 2021, Fu et al., 2022)).
• Iterative Mask Scheduling: Decoding or entropy coding schedules decode tokens in several rounds by decreasing the mask ratio (e.g., QLDS [Mentzer et al.], applied in NeuralMDC (Hu et al., 2024); cosine or polynomial scheduling in MaskViT (Gupta et al., 2022) and MaskINT (Ma et al., 2023)).

Objectives are paired with masking and include pixel-level $\ell_2$ or $\ell_1$ regression on masked reconstructions (VideoMAE, OmniMAE, RVM), discrete token classification with cross-entropy (VQ or codebook tokens in BEVT, VIOLET, MaskViT), masked token prediction distributions for entropy coding (NeuralMDC), or denoising objectives in latent space (VDT, Mask$^2$DiT).

3. Applications: Self-Supervised Pretraining, Compression, and Generation

Masked video transformers provide a unified interface for a diverse range of video tasks:

• Self-supervised Pretraining for Video Understanding:
  • VideoMAE (Tong et al., 2022), OmniMAE (Girdhar et al., 2022), and BEVT (Wang et al., 2021) demonstrate that high-ratio masked reconstruction pretraining on video is effective for downstream action recognition, with VideoMAE reaching 87.4% on Kinetics-400 using only in-domain data.
  • Data Collection-free Masked Video Modeling (Ishikawa et al., 2024) shows that pseudo-motion augmentations of static images coupled with masked video modeling can close the gap to real-video pretraining.
  • MAM² (Song et al., 2022) disentangles appearance and motion prediction via masking and dual decoders, yielding faster convergence and competitive accuracy to VideoMAE with fewer epochs.
  • VIOLET (Fu et al., 2021, Fu et al., 2022) and BEVT (Wang et al., 2021) demonstrate that masked video transformers improve transferability in video-language and spatio-temporal tasks, especially when integrating high-level semantic targets (SIF from Swin-B (Fu et al., 2022)).
• Video Compression and Loss-Resilient Delivery:
  • NeuralMDC (Hu et al., 2024) employs a bidirectional masked transformer as a learned entropy model for multiple description coding, enabling independent entropy coding of descriptions, robust recovery under packet loss, and state-of-the-art loss resilience.
  • Masked token prediction at arbitrary spatial and temporal positions allows the transformer to operate as a universal entropy model capable of filling in lost (masked) tokens using context.
• Video Generation, Prediction, and Editing:
  • VDT (Lu et al., 2023) leverages a unified spatial-temporal mask modeling mechanism within a pure-transformer diffusion framework to support unconditional generation, conditional prediction, interpolation, and inpainting, with the mask pattern defining the generative task.
  • Mask$^2$DiT (Qi et al., 25 Mar 2025) introduces dual masks (scene/segment assignment and conditional masks) in a DiT backbone to enforce fine-grained text-to-video segment alignment and enable autoregressive scene extension.
  • MaskViT (Gupta et al., 2022) and MaskINT (Ma et al., 2023) employ iterative non-autoregressive decoding via mask scheduling for fast generative modeling and video editing, respectively, with orders-of-magnitude inference speedups over diffusion models.
• Video Inpainting and Super-Resolution:
  • DMT (Yu et al., 2023) applies masking to select valid spatiotemporal tokens, using deficiency-aware self-attention and receptive field contextualization to inpaint missing regions across frames, outperforming prior SOTA on DAVIS and YouTube-VOS.
  • MIA-VSR (Zhou et al., 2024) couples inter/intra-frame attention with adaptive block-wise mask prediction, learning to propagate only salient, high-change features for computationally efficient video super-resolution.

4. Empirical Findings, Ablations, and Performance

Empirical studies across these works converge on several key findings:

• Masking Ratio and Pattern:
  • Extremely high mask ratios (90–95%) are viable in video because of temporal redundancy, leading to efficient pretraining (VideoMAE, OmniMAE). Tube/block masking further encourages modeling of temporal dependencies.
  • Adaptive/motion-aware masking effectively reduces compute while preserving or enhancing accuracy at high temporal resolutions (MGTC (Feng et al., 2024), MIA-VSR (Zhou et al., 2024)).
• Benefits of Masked Prediction:
  • For video-language tasks, masked visual modeling improves video question answering and retrieval across VIOLET, BEVT, and VIOLETv2 (Fu et al., 2022). In VIOLETv2, high-level semantic reconstruction targets dominate pixel- or gradient-level targets.
  • In video compression, bidirectional masked transformers used as entropy models (NeuralMDC) provide 2–8× slower degradation under random loss compared to existing neural codecs, with modest (10–20%) bit-rate overhead when using multiple descriptions (Hu et al., 2024).
  • Masked video transformers enable efficient long-sequence modeling: recurrent masked autoencoders (RVM (Zoran et al., 15 Dec 2025)) achieve up to 30× greater parameter efficiency than prior video MAEs while maintaining SOTA accuracy and stable feature propagation.
• Ablations and Best Practices:
  • Decoupling appearance (VQ token/codebook prediction) and motion (RGB-difference or optical flow) pretext tasks in MAM² yields faster convergence and improved generalization (Song et al., 2022).
  • Joint random+blockwise masking surpasses attention-based masking for video-language fusion in VIOLETv2 (Fu et al., 2022).
  • For generation, symmetric dual-masking and segment-level conditioning (Mask$^2$DiT) optimize semantic and visual alignment across long videos or multiple scenes (Qi et al., 25 Mar 2025).

5. Limitations, Extensions, and Future Directions

Despite their versatility, several challenges and opportunities persist:

• Model Complexity and Scalability: Full spatio-temporal attention scales quadratically with frame/token number. Efficiency-oriented designs (motion-guided masking, blockwise attention, recurrence (Zoran et al., 15 Dec 2025)) are required for very long or high-resolution sequences.
• Masking Schedules and Adaptivity: Most models use fixed or heuristic scheduling for masking; learned or task-adaptive scheduling remains underexplored.
• Generative Fidelity and Temporal Coherence: While diffusion and masked generation models enable complex tasks (animation, inpainting, scene generation), consistent frame-level structure over long ranges is not always guaranteed. Global video-visual token connectivity, as in Mask$^2$DiT, and hierarchical or autoregressive designs offer potential solutions.
• Domain Transfer and Data Efficiency: Synthetic data augmentation (pseudo-motion from images) (Ishikawa et al., 2024) or transfer from image-pretrained backbones (BEVT, VIOLETv2) are promising for data-limited regimes but may not fully substitute for real video exposure, especially in dynamic or compositional domains.
• Applications Beyond Core Modeling: Masked video transformers now underpin state-of-the-art in robust compression (Hu et al., 2024), general-purpose video generation (Lu et al., 2023), video-language understanding (Fu et al., 2021, Fu et al., 2022), and efficient edge inference (MIA-VSR).

6. Summary Table: Key Masked Video Transformer Approaches

Approach	Masking Strategy	Application
VideoMAE (Tong et al., 2022)	Tube/block, 90–95% mask	Self-supervised pretrain; video recognition
BEVT (Wang et al., 2021)	Block-tube for video stream	Spatio-temporal representation learning
NeuralMDC (Hu et al., 2024)	Random mask; bidirectional context	Loss-resilient neural video codec
VDT (Lu et al., 2023)	Unified spatio-temporal masking	Diffusion-based video generation, completion
MaskINT (Ma et al., 2023)	Masked token interpolation	Fast video editing, frame interpolation
MAM² (Song et al., 2022)	Tube masking; disentangled losses	Joint appearance-motion pretraining
DMT (Yu et al., 2023)	Spatiotemporal mask activation	Video inpainting, deficiency-aware modeling
VIOLETv2 (Fu et al., 2022)	Random+blockwise masking	Video-language modeling; QA, retrieval
RVM (Zoran et al., 15 Dec 2025)	Heavy spatial masking, rec. encoder	Efficient general-purpose video representation
MGTC (Feng et al., 2024)	Motion-var. token compression	Compute-efficient recognition
Mask$^2$DiT (Qi et al., 25 Mar 2025)	Dual symmetric/conditional masking	Multi-scene text-aligned video generation

The further development of masked video transformers is anticipated to drive advances not only in standard video understanding and generation benchmarks, but also in real-world domains such as robust streaming, large-scale video synthesis, embodied AI, and cross-modal video-language alignment. The integration of adaptive masking, hybrid transformer-recurrent and memory-augmented mechanisms, and data-efficient transfer learning are prominent directions motivated by current empirical evidence.