Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator

Abstract: Unified multimodal models integrating visual understanding and generation face a fundamental challenge: visual generation incurs substantially higher computational costs than understanding, particularly for video. This imbalance motivates us to invert the conventional paradigm: rather than extending understanding-centric MLLMs to support generation, we propose Uni-ViGU, a framework that unifies video generation and understanding by extending a video generator as the foundation. We introduce a unified flow method that performs continuous flow matching for video and discrete flow matching for text within a single process, enabling coherent multimodal generation. We further propose a modality-driven MoE-based framework that augments Transformer blocks with lightweight layers for text generation while preserving generative priors. To repurpose generation knowledge for understanding, we design a bidirectional training mechanism with two stages: Knowledge Recall reconstructs input prompts to leverage learned text-video correspondences, while Capability Refinement fine-tunes on detailed captions to establish discriminative shared representations. Experiments demonstrate that Uni-ViGU achieves competitive performance on both video generation and understanding, validating generation-centric architectures as a scalable path toward unified multimodal intelligence. Project Page and Code: https://fr0zencrane.github.io/uni-vigu-page/.

• The paper demonstrates that a unified diffusion-based framework effectively couples video generation and understanding via a bidirectional uni-flow process.
• It leverages continuous and discrete flow matching within a shared Transformer backbone enhanced with modality-specific lightweight FFNs.
• Experimental results show competitive performance with reduced computational requirements, enabling scalable multimodal video-language learning.

Uni-ViGU: A Unified Diffusion-Based Framework for Video Generation and Understanding

Introduction and Context

The pursuit of unified multimodal architectures has been characterized by attempts to jointly optimize visual understanding and generation within a single computational framework. While previous approaches have predominantly extended Multimodal LLMs (MLLMs) with generation capacity, such models face critical inefficiencies, particularly when scaled to video, due to the dramatically higher computational cost of generation vs. understanding. "Uni-ViGU: Towards Unified Video Generation and Understanding via A Diffusion-Based Video Generator" (2604.08121) addresses this challenge by inverting the paradigm: instead of augmenting MLLMs, the architecture extends the capabilities of a pretrained diffusion-based video generator to serve both generation and understanding. This approach leverages the strong generative priors and rich multimodal semantic alignments acquired during large-scale text-to-video pretraining as the backbone for efficient bidirectional multimodal processing.

Unified Diffusion-Based Framework

The core technical innovation of Uni-ViGU is the formulation and implementation of a unified flow matching process—termed "uni-flow"—that enables coherent text and video generation within a single, tightly coupled Transformer-based backbone.

Traditionally, video generation via diffusion models operates in the continuous latent space of a VAE, utilizing ODE-based denoising from Gaussian noise. In contrast, text generation is inherently discrete and usually modeled in the autoregressive token prediction paradigm. Uni-ViGU bridges this modality gap through:

• Continuous flow matching for video: latent video representations are constructed via linear interpolation between noise and encoded video latents, and the model is trained to predict the transport velocity through the latent space per reconstructed time step.
• Discrete flow matching for text: following advances in discrete diffusion modeling, token embeddings are used as the target for flow matching in a continuous vector space, despite their discrete origin. Generation is refined by decoding denoised text embeddings to tokens via maximum similarity over the learned embedding matrix.

A distinctive feature is the independent sampling of flow integration times for each modality, enabling learning of cross-modal dependencies even when one modality is partially denoised and the other is still noisy, thus fostering deeper representational alignment.

Figure 1: Overview of the Uni-ViGU uni-flow process, unifying video (continuous flow matching) and text (discrete flow matching) via a single shared Transformer backbone with modality-specific FFN branches.

Modality-Driven Mixture-of-Experts Design

A nuanced analysis of the Diffusion Transformer (DiT) architecture underpins Uni-ViGU's mixture-of-experts (MoE) solution. Empirical and theoretical considerations indicate that cross-modal semantic alignment emerges primarily through the relational capacity of self-attention and cross-attention layers. In contrast, domain- and modality-specific generative priors are encoded in the feed-forward networks (FFNs).

Consequently, Uni-ViGU:

• Shares all attention layers across modalities to maximize the reuse and transfer of cross-modal knowledge accumulated in the pretrained video generator.
• Introduces modality-specific, lightweight FFN branches for text and video. The video FFN branch is initialized from pretrained weights, while the text FFN branch is trained anew, facilitating efficient adaptation to the discrete text modality with minimal parameter overhead.

This structured MoE strategy enables deterministic routing of activations based on modality identity, allowing the shared Transformer backbone to learn cross-modal dependencies while retaining domain specialization.

Bidirectional Training and Knowledge Transfer

The paper proposes a two-stage, bidirectional training regime:

• Stage 1: Knowledge Recall. The model reconstructs the input prompt as the target text, frequently dropping the prompt as a conditioning signal to prevent shortcut learning. The optimization emphasizes the text branch (due to token count imbalance) to expedite the adaptation of the pretrained generative backbone for text prediction.
• Stage 2: Capability Refinement. The text prediction target is replaced with a detailed video caption, demanding fine-grained visual understanding. This forces the shared attention layers to extract semantic details from the video representation for caption generation, thus establishing bidirectional, discriminative, and deeply aligned multimodal representations.

Inference procedures are symmetric for both video generation (starting from noisy video latent, conditioned on clean text) and video understanding (starting from clean video latent, generating text from noise), and enable joint multimodal generation by simultaneously denoising both modalities, allowing mutual guidance via cross-attention.

Experimental Validation

Uni-ViGU is initialized from a state-of-the-art text-to-video diffusion generator (Wan2.1) and trained on curated video-prompt and video-prompt-caption datasets. The results reveal:

• The model demonstrates effective joint video-text generation, producing temporally coherent videos and detailed, semantically faithful captions that are significantly more descriptive than input prompts.
• During joint denoising, the emerging video and text latents provide reciprocal semantic and visual guidance, manifesting in high-quality, mutually consistent multimodal outputs.

  Figure 2: Qualitative video-text joint generation: the model co-evolves videos and captions, leveraging reciprocal semantic and visual guidance.

Numerical results cited in the paper demonstrate that, with only a week of training on 16 H800 GPUs, the model achieves competitive performance relative to dual-tower or MLLM-centric approaches, yet with dramatically reduced computational requirements for unification and inference.

Implications and Future Directions

Uni-ViGU’s generation-centric paradigm makes several strong claims:

• Repurposed generation knowledge enables scalable unified video-text understanding and generation without the computational overhead of duplicated or dual-branch architectures.
• Shared attention with modality-specific FFNs is both sufficient and parameter-efficient for cross-modal transfer.
• Bidirectional training through prompt reconstruction and caption refinement is effective for activating generative priors in the reverse direction for understanding.

Practically, this paradigm unlocks unified multimodal learning that is scalable to videos, addressing the prohibitive cost asymmetry of typical MLLM-augmented diffusion models. Theoretically, it supports the hypothesis that generative modeling establishes a strong foundation for multimodal intelligence, analogous to human perceptual and linguistic development.

Potential future directions include:

• Scaling to larger video-text corpora and high-resolution, long-duration video.
• Incorporating more sophisticated MoE routing mechanisms for further specialization.
• Extending the uni-flow framework to additional modalities (e.g., audio, 3D data).
• Exploring unified frameworks for editing, retrieval, and interaction tasks.

Conclusion

Uni-ViGU introduces a principled, computationally efficient approach to the unification of video generation and understanding by leveraging generative diffusion transformers as the foundation. The uni-flow formulation, coupled with a modality-driven MoE architecture and bidirectional cross-modal training, demonstrates that pretrained video generative models can be effectively and efficiently repurposed for bidirectional multimodal intelligence, setting a practical path forward for scalable unified video-language systems.