MotionCraft: Physics-based Zero-Shot Video Generation

Abstract: Generating videos with realistic and physically plausible motion is one of the main recent challenges in computer vision. While diffusion models are achieving compelling results in image generation, video diffusion models are limited by heavy training and huge models, resulting in videos that are still biased to the training dataset. In this work we propose MotionCraft, a new zero-shot video generator to craft physics-based and realistic videos. MotionCraft is able to warp the noise latent space of an image diffusion model, such as Stable Diffusion, by applying an optical flow derived from a physics simulation. We show that warping the noise latent space results in coherent application of the desired motion while allowing the model to generate missing elements consistent with the scene evolution, which would otherwise result in artefacts or missing content if the flow was applied in the pixel space. We compare our method with the state-of-the-art Text2Video-Zero reporting qualitative and quantitative improvements, demonstrating the effectiveness of our approach to generate videos with finely-prescribed complex motion dynamics. Project page: https://mezzelfo.github.io/MotionCraft/

• The paper introduces a zero-shot approach that leverages physics-based optical flows to animate static images with realistic, coherent motion.
• The method integrates pretrained image diffusion models with optical flows from physics simulations, preserving visual integrity and dynamics.
• Experimental results show superior performance over T2V0, demonstrating enhanced temporal consistency and adaptability across diverse motion scenarios.

MotionCraft: Physics-based Zero-Shot Video Generation

MotionCraft represents an innovative approach in the field of video generation by leveraging a physics-based zero-shot technique, allowing the synthesis of videos with realistic and physically plausible motion without the need for extensive model training or finetuning.

Introduction

MotionCraft capitalizes on the capabilities of pretrained still image diffusion models, such as Stable Diffusion, and integrates them with optical flow derived from physics simulations. The core idea is to warp the noise latent space within the image diffusion model using these optical flows, capturing complex dynamics in a manner that seamlessly animates an image over time. The optical flow encapsulates a physical description of motion, enabling the generation of videos where motion dynamics are consistent and aligned with physical principles. This approach contrasts with traditional pixel-space applications, avoiding issues such as content displacement and pixel artefacts.

Figure 1: MotionCraft overview. A video is generated from a starting image using a pretrained still image generative model by warping noise latents according to an optical flow description of the motion to be synthesised.

Methodology

MotionCraft operates by first encoding an initial image using a VQ-VAE embedded within the diffusion model to obtain a latent representation. This latent representation is then iteratively transformed using a sequence of optical flows derived from a physics simulator. Each transformation step involves applying these flows within the latent space, thus preserving the structural and textural integrity of the visual content while also ensuring coherent motion dynamics. A key observation underpinning this method is the correlation between optical flow in image space and the latent space of diffusion models, as demonstrated by experiments conducted on the MSU Video Frame Interpolation Benchmark dataset.

Experimental Results

MotionCraft was evaluated against the state-of-the-art zero-shot video generator, Text2Video-Zero (T2V0), showcasing its superior ability to maintain temporal consistency and generate plausible new content. Experiments spanned across various scenarios including fluid dynamics, rigid body motion, and multi-agent systems, effectively demonstrating the versatility and robustness of the model. Notably, the method excels in synthesizing realistic content complemented by consistent illumination and motion dynamics without the traditional requirement for extensive data or computational resources.

Figure 2: Multi-agent system simulation: bird flock. Top: MotionCraft; Bottom: T2V0.

Key Contributions and Ablations

Core enhancements such as the Multiple Cross-Frame Attention (MCFA) mechanism and Spatial-$\eta$ weighting played pivotal roles in improving video coherence and content generation. Ablation studies provided insights into each component's contribution, highlighting that attending to both the initial frame and recent history is crucial for consistency and quality. The method's novel approach of employing spatially-varying noise during the diffusion process further refines the generated outputs, especially when novel content is introduced into the frames.

Limitations and Future Directions

While MotionCraft achieves remarkable outcomes, it is inherently limited by the base image generation model's capabilities and the specificity of the optical flow derived from physical simulations. Opportunities for future research include the integration of generative optical flow models conditioned on both initial frames and prompts, as well as enhancements to the interaction between generative models and physics simulators to further tighten the feedback loop, ensuring even higher levels of physical fidelity in video outputs.

Conclusion

MotionCraft offers a robust, zero-shot framework for video generation that leverages existing image diffusion models supplemented by optical flows driven by physical simulations. Through careful methodological design and innovation, it successfully generates high-fidelity videos across a variety of dynamic scenarios, representing a significant step forward in the synthesis of visually and physically coherent video content. The method's ability to seamlessly blend physics with visual generation offers broad implications for fields where video representation of complex dynamics is critical.