HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation

Abstract: Customized video generation aims to produce videos featuring specific subjects under flexible user-defined conditions, yet existing methods often struggle with identity consistency and limited input modalities. In this paper, we propose HunyuanCustom, a multi-modal customized video generation framework that emphasizes subject consistency while supporting image, audio, video, and text conditions. Built upon HunyuanVideo, our model first addresses the image-text conditioned generation task by introducing a text-image fusion module based on LLaVA for enhanced multi-modal understanding, along with an image ID enhancement module that leverages temporal concatenation to reinforce identity features across frames. To enable audio- and video-conditioned generation, we further propose modality-specific condition injection mechanisms: an AudioNet module that achieves hierarchical alignment via spatial cross-attention, and a video-driven injection module that integrates latent-compressed conditional video through a patchify-based feature-alignment network. Extensive experiments on single- and multi-subject scenarios demonstrate that HunyuanCustom significantly outperforms state-of-the-art open- and closed-source methods in terms of ID consistency, realism, and text-video alignment. Moreover, we validate its robustness across downstream tasks, including audio and video-driven customized video generation. Our results highlight the effectiveness of multi-modal conditioning and identity-preserving strategies in advancing controllable video generation. All the code and models are available at https://hunyuancustom.github.io.

An Overview of "HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation"

The paper "HunyuanCustom: A Multimodal-Driven Architecture for Customized Video Generation" proposes a novel framework aimed at enhancing the precision and identity consistency of customized video generation. Addressing the limitations of previous methods, such as inconsistent identity portrayal and restricted input modalities, the researchers present HunyuanCustom, a comprehensive multi-modal video generation model designed to operate with high identity fidelity across diverse contexts.

Key Components and Methodology

Centered around a unique architecture, HunyuanCustom stands out with its integration of multiple input modalities—text, images, audio, and video—enabling it to deliver more robust and customizable video outputs. Built on the foundational HunyuanVideo model, the framework introduces several critical enhancements that contribute to its advanced capabilities:

1. Text-Image Fusion Module: Leveraging LLaVA technology, this component facilitates the seamless integration of textual and image inputs, bolstering the model's multi-modal comprehension and enabling nuanced video generation that reflects precise identity characteristics.

2. Image ID Enhancement Module: This module uses temporal concatenation to ensure identity consistency across video frames, thereby maintaining the integrity of the subject's appearance throughout the video duration.

3. AudioNet Module: Designed for audio-conditioned generation, it employs spatial cross-attention for hierarchical alignment between audio and video features, thus harmonizing auditory inputs with corresponding visual dynamics.

4. Video-Driven Injection Module: This component utilizes a patchify-based feature-alignment network to efficiently transcode video inputs, integrating compressed conditional video content into the model's latent space without compromising computational efficiency.

Experimental Validation and Results

Through a series of rigorous experiments involving both single- and multi-subject scenarios, HunyuanCustom demonstrated a marked improvement over existing methods—both open-source and proprietary—regarding ID consistency, realism, and text-video alignment. This was quantitatively validated using advanced evaluation metrics such as ID consistency (Arcface similarity), text-video alignment (CLIP-B), subject similarity (DINO-Sim), temporal consistency, and dynamic degree measurements.

The results underscore the framework's robustness across various downstream applications, including audio-driven and video-driven customized video generation. As such, HunyuanCustom implicates numerous practical applications, notably in virtual human advertisement, virtual try-ons, and detailed video editing, reflecting its versatility and adaptability in real-world scenarios.

Implications and Future Directions

HunyuanCustom addresses critical challenges in controllable video generation by integrating robust identity-preserving strategies with multi-modal conditioning. Its successful design and implementation can potentially pave the way for future advancements in artificial intelligence-generated content (AIGC), particularly when applied to dynamic and customizable video contexts. The availability of its code and models facilitates the replication and extension of this framework, suggesting avenues for further research into fine-grained customization techniques that could expand multimodal generative models' capabilities.

In conclusion, HunyuanCustom represents a significant progression in the field of customized video generation, offering a powerful solution that bridges gaps in identity consistency and multi-modal integration, thus enhancing the potential for precision-tailored video production in various domains.