OmniForcing: Unleashing Real-time Joint Audio-Visual Generation

Abstract: Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency due to their bidirectional attention dependencies, hindering real-time applications. We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autoregressive generator. However, naively applying causal distillation to such dual-stream architectures triggers severe training instability, due to the extreme temporal asymmetry between modalities and the resulting token sparsity. We address the inherent information density gap by introducing an Asymmetric Block-Causal Alignment with a zero-truncation Global Prefix that prevents multi-modal synchronization drift. The gradient explosion caused by extreme audio token sparsity during the causal shift is further resolved through an Audio Sink Token mechanism equipped with an Identity RoPE constraint. Finally, a Joint Self-Forcing Distillation paradigm enables the model to dynamically self-correct cumulative cross-modal errors from exposure bias during long rollouts. Empowered by a modality-independent rolling KV-cache inference scheme, OmniForcing achieves state-of-the-art streaming generation at $\sim$25 FPS on a single GPU, maintaining multi-modal synchronization and visual quality on par with the bidirectional teacher.\textbf{Project Page:} \href{https://omniforcing.com}{https://omniforcing.com}

• The paper introduces a three-stage distillation pipeline that converts a bidirectional teacher into a causal autoregressive generator for real-time audio-visual synthesis.
• It incorporates asymmetric block-causal alignment and learnable audio sink tokens to synchronize different audio and video sampling rates effectively.
• Experiments reveal a 35x speedup in inference with near-parity in fidelity and synchrony, supporting deployment in interactive applications.

OmniForcing: Real-Time Streaming Joint Audio-Visual Generation via Causal Diffusion Distillation

Introduction and Motivation

High-fidelity joint audio-visual generation using large diffusion-based models has attained substantial progress in audio-video synchrony, temporal consistency, and perceptual quality by leveraging fully bidirectional attention over both streams. However, such models—exemplified by LTX-2—are fundamentally encumbered by large inference latency due to the necessity of processing entire sequences simultaneously. This quadratic complexity in both sequence length and modality dimension precludes their applicability in interactive or real-time use cases. OmniForcing (2603.11647) addresses this gap by introducing a distillation pipeline and architectural innovations to convert a powerful, offline bidirectional teacher into a high-fidelity streaming autoregressive generator that functions at real-time frame rates, with negligible compromise in temporal or perceptual coherence.

OmniForcing's principal contribution lies in resolving the architectural and optimization challenges brought by the severe temporal asymmetry between modalities (video at 3 FPS, audio at 25 FPS) and the token density mismatch. This is achieved through a tailored block-causal alignment, architectural stabilizers, and a three-stage distillation pipeline.

Figure 1: OmniForcing achieves real-time streaming ($\sim$25 FPS) with low TTFC ($\sim$0.7s), dramatically outperforming the bidirectional teacher’s latency while maintaining comparable fidelity.

Technical Innovations

Three-Stage Distillation Pipeline

OmniForcing employs a structured, three-stage distillation process that mitigates conditional distributional shifts and stabilizes training when transitioning from bidirectional to causal inference. The pipeline consists of:

1. Distribution Matching Distillation (DMD): The model is adapted for few-step denoising while retaining bidirectional receptive fields, yielding a tractable teacher trajectory for subsequent causal adaptation.
2. Causal ODE Regression: With block-causal masking in place, the model is regressed against ODE trajectories of the Stage I teacher. The asymmetric masking accounts for modality-specific latency and stride properties of the underlying VAEs.
3. Joint Self-Forcing Distillation: Exposure bias and drift accumulation, particularly acute in long multimodal sequences, are addressed by autoregressively unrolling and training on the model’s own predictions, enforcing strict synchrony and error-correction across modalities.

   Figure 2: The OmniForcing three-stage pipeline, highlighting sequential adaptation—first for fast denoising, then causal masking, finally self-forcing for exposure-bias mitigation.

Asymmetric Block-Causal Alignment and Attention Masking

Standard frame-level causal masking is inadequate in the presence of a non-integral, high-ratio mismatch between audio and video sampling rates. OmniForcing introduces macro-block alignment tied to physical one-second windows, synchronizing exactly 3 video and 25 audio latent frames per block—this design harmonizes attention masking across depth, supporting temporal coherence and perfect synchronization during autoregressive generation.

A global prefix block at the sequence origin consolidates modality-specific initial latents and acts as a semantic anchor visible to all subsequent blocks, enhancing the transfer of bidirectional context into the constrained autoregressive setting.

Figure 3: Visualization of the asymmetric block-causal mask; blue areas allow attention within macro-blocks and global prefix, while white areas enforce strict causality and prevent leakage.

Audio Sink Tokens and Identity RoPE Stabilization

Applying causal masks to highly sparse, early audio blocks can precipitate numerical instability, gradient explosion, and Softmax collapse due to low denominator entropy. To counteract this, the method integrates learnable Audio Sink Tokens—embedded at the start of the audio sequence and fixed in the prefix block—that serve both as entropy reservoirs and memory buffers. An Identity RoPE constraint eliminates spurious position encoding on these sink tokens, maintaining their invariance irrespective of sequence index and precluding positional interference.

Alternative stabilizers, such as QK-Norm and Tanh-Gated mechanisms, were empirically found to be less effective or even destructive, as shown by higher denoising losses and visual artifacts.

Experimental Evaluation

The efficacy of OmniForcing is demonstrated on JavisBench and VBench using multiple metrics for audio-visual quality, text-alignment, cross-modal consistency, and timing synchrony. Notable outcomes include:

• Inference Efficiency: End-to-end generation throughput of ~25 FPS with TTFC under 0.7s for 480p, compared to 197s for the bidirectional teacher (a ~35x speedup).
• Fidelity and Synchrony: Near-parity with the teacher is observed in FVD (137.2 vs. 125.4), FAD (5.7 vs. 4.6), and DeSync (0.392 vs. 0.384), with only modest declines in the strictest temporal metrics.
• Distillation Robustness: On VBench, OmniForcing marginally surpasses the teacher in per-frame metrics including aesthetic and imaging quality, echoing earlier findings from DMD-based student models.

Qualitative analysis corroborates these findings, with generated clips exhibiting both perceptually rich audio textures and precisely aligned video events across diverse, complex scenarios.

Figure 4: Qualitative samples demonstrate tightly synchronized and contextually coherent audio-visual content in challenging multi-source acoustic and visual scenarios.

Practical and Theoretical Implications

OmniForcing unlocks a previously impractical regime for audio-visual generative models—interactive, real-time, low-latency inference on commodity GPUs. This positions the architecture for deployment in media production, virtual communication, and real-time simulation environments. The block-causal alignment and audio sink token paradigm provide a transferable blueprint for other multi-rate, multi-modal causal modeling tasks.

Theoretically, the results affirm that the trade-off between bidirectional context and real-time performance can be tightly controlled with distribution-matching distillation, explicit causal factorization, and careful architectural stabilization. The strong preservation of multi-modal synchrony and content quality—despite strict causality—suggests the intrinsic redundancy in cross-modal bidirectional processing can be successfully captured and transferred into an efficient autoregressive generative process.

Future Perspectives

Immediate avenues include further scaling of the architecture via asymmetric tensor parallelism and extension to higher resolutions and longer sequences. The modality-agnostic rolling KV-cache architecture offers a practical path to more complex multi-modal or multi-stream synchronization tasks. More broadly, the architectural stabilization strategies—sink tokens and macro-block alignment—may generalize to large-scale streaming models beyond audio-visual domains, such as multi-sensor fusion and temporal language grounding.

Conclusion

OmniForcing presents a comprehensive solution to low-latency, high-fidelity joint audio-visual generation by distilling a powerful bidirectional teacher into a real-time, block-causal autoregressive framework. Its integration of macro-block causal masking, audio sink token stabilization, self-forcing error correction, and a modality-independent rolling KV-cache achieves a decisive advance in streaming multi-modal generation, with extensive experimental validation establishing its state-of-the-art efficiency and output quality (2603.11647).