Magic 1-For-1: Generating One Minute Video Clips within One Minute

Abstract: In this technical report, we present Magic 1-For-1 (Magic141), an efficient video generation model with optimized memory consumption and inference latency. The key idea is simple: factorize the text-to-video generation task into two separate easier tasks for diffusion step distillation, namely text-to-image generation and image-to-video generation. We verify that with the same optimization algorithm, the image-to-video task is indeed easier to converge over the text-to-video task. We also explore a bag of optimization tricks to reduce the computational cost of training the image-to-video (I2V) models from three aspects: 1) model convergence speedup by using a multi-modal prior condition injection; 2) inference latency speed up by applying an adversarial step distillation, and 3) inference memory cost optimization with parameter sparsification. With those techniques, we are able to generate 5-second video clips within 3 seconds. By applying a test time sliding window, we are able to generate a minute-long video within one minute with significantly improved visual quality and motion dynamics, spending less than 1 second for generating 1 second video clips on average. We conduct a series of preliminary explorations to find out the optimal tradeoff between computational cost and video quality during diffusion step distillation and hope this could be a good foundation model for open-source explorations. The code and the model weights are available at https://github.com/DA-Group-PKU/Magic-1-For-1.

• The paper introduces Magic 1-For-1, an efficient video generation framework that decomposes the text-to-video task and employs dual distillation to generate one minute of video in one minute.
• The framework achieves significant speed improvements, generating one minute of video in one minute and reducing model memory footprint from 32GB to 16GB using int8 quantization.
• Key technical innovations include multi-modal guidance using retrieved images, diffusion step distillation via a dual distillation paradigm, and parameter efficiency techniques.

The paper introduces an efficient video generation framework by decomposing the text-to-video task into two sequential subtasks: text-to-image and image-to-video. This factorization allows the authors to leverage the well-established body of research on text-to-image diffusion models and then refine these results for video synthesis, ultimately reducing the number of diffusion steps required.

The core contributions and technical innovations include:

Task Factorization and Multi-Modal Guidance

• The approach first generates high-fidelity images from text prompts and, subsequently, synthesizes corresponding video frames by treating the image as the initial frame.
• A multi-modal guidance mechanism is introduced by augmenting the text encoder with visual input. The retrieved reference images are embedded via a vision LLM (VLM) and concatenated with text embeddings to reinforce semantic coherence, thereby improving temporal consistency in generated videos.

Diffusion Step Distillation via Dual Distillation Paradigm

• The diffusion step distillation framework is formulated as a dual distillation process that combines both step distillation and classifier-free guidance (CFG) distillation.
• The method leverages a state-of-the-art distribution matching approach (DMD2) where three models are coordinated: a few‐step generator, a “real” video model approximating the true data distribution, and a “fake” video model aligned to the generated distribution.
• The distribution matching signal is obtained via aligning gradients computed from the prediction of noise at each step. In one formulation, if we denote the latent at time step $t$ by $\mathbf{z}_t$ and the noise schedule by $\sigma_t$, then the noise prediction updates can be expressed as
  • $y$ represents the text prompt,
  • $\mathcal{R}(y)$ denotes the set of retrieved relevant images, and
  • $\epsilon_\theta$ is the learned noise prediction function.
• The paper further introduces CFG distillation to eliminate the iterative guidance computation during inference by training a student model to directly output the guided predictions using an additional loss term.

Parameter Efficiency and Quantization

• The authors employ parameter sparsification techniques, including leveraging LoRA for efficient parameter updates in the fake model branch during DMD2 training, ensuring stable gradient flow when the data distributions between the pre-trained and few-step generators differ.
• For inference, an int8 weight-only quantization strategy is applied to the denoising network (covering transformer blocks and VLM encoder), effectively reducing the model’s memory footprint from approximately 32GB to 16GB while keeping peak runtime memory around 30GB.
• The quantization process involves scaling the original bfloat16 weights into int8 representation, for example using a transformation of the form $$w_{\text{int8}} = \text{round} \left( w_{\text{bf16}} \times \frac{127}{\max(|w_{\text{bf16}}|)} \right),$$ with $w_{\text{bf16}}$ being the original weight value and $\max(|w_{\text{bf16}}|)$ the maximal absolute value in the weight tensor.

Efficiency and Performance Benchmarking

• Using a test-time sliding window strategy, the model is demonstrated to generate a 5-second video clip in 3 seconds and, overall, a one-minute video in one minute (i.e., about 1 second per video second on average), which indicates significant improvements in inference latency compared to standard diffusion models that require hundreds of iterative steps.
• Empirical evaluations are performed on both a customized VBench for portrait video synthesis and a General VBench for generic video synthesis and include standard quantitative metrics such as FID, FVD, and LPIPS.
• Ablation studies reveal that the optimal configuration for the few-step generator is achieved with a 4-step distillation on the image-to-video generation task. Notably, it is observed that step distillation converges much faster on TI2V tasks than on the full text-to-video synthesis pipeline—a result that is supported by detailed training loss curves and metric evaluations across temporal consistency and motion dynamics.

Empirical Insights and Comparative Evaluations

• The performance comparison with several state-of-the-art text-to-image-to-video models demonstrates that the proposed method not only achieves superior trade-offs between visual fidelity, temporal coherence, and computational efficiency but also shows robustness across multiple evaluation dimensions such as subject consistency, motion smoothness, and aesthetic quality.
• The experiments emphasize that the diffusion step distillation, when paired with multi-modal inputs and quantization, effectively mitigates biases inherent in the base model and markedly improves generation quality even with a reduced sampling budget.
• Training is carried out on 128 GPUs over two weeks using a diverse dataset aggregated from sources including WebVid-10M, Panda-70M, and Koala-36M, ensuring the learned generative prior is both rich and robust.

In summary, the paper presents a methodologically rigorous framework that leverages task decomposition, an efficient dual distillation scheme, and quantization to significantly accelerate video generation while minimizing degradation in visual and motion quality. This work contributes to making diffusion-based video synthesis more practical for real-world applications by reducing both inference latency and memory consumption without sacrificing high-fidelity output.