V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

Abstract: A major challenge for modern AI is to learn to understand the world and learn to act largely by observation. This paper explores a self-supervised approach that combines internet-scale video data with a small amount of interaction data (robot trajectories), to develop models capable of understanding, predicting, and planning in the physical world. We first pre-train an action-free joint-embedding-predictive architecture, V-JEPA 2, on a video and image dataset comprising over 1 million hours of internet video. V-JEPA 2 achieves strong performance on motion understanding (77.3 top-1 accuracy on Something-Something v2) and state-of-the-art performance on human action anticipation (39.7 recall-at-5 on Epic-Kitchens-100) surpassing previous task-specific models. Additionally, after aligning V-JEPA 2 with a LLM, we demonstrate state-of-the-art performance on multiple video question-answering tasks at the 8 billion parameter scale (e.g., 84.0 on PerceptionTest, 76.9 on TempCompass). Finally, we show how self-supervised learning can be applied to robotic planning tasks by post-training a latent action-conditioned world model, V-JEPA 2-AC, using less than 62 hours of unlabeled robot videos from the Droid dataset. We deploy V-JEPA 2-AC zero-shot on Franka arms in two different labs and enable picking and placing of objects using planning with image goals. Notably, this is achieved without collecting any data from the robots in these environments, and without any task-specific training or reward. This work demonstrates how self-supervised learning from web-scale data and a small amount of robot interaction data can yield a world model capable of planning in the physical world.

• The paper introduces a self-supervised video framework that pre-trains on over 1M hours of video data to capture detailed world dynamics.
• It employs a joint-embedding predictive architecture with vision transformers to achieve state-of-the-art performance in motion understanding and human action anticipation.
• The action-conditioned model enables effective zero-shot planning for robotic tasks like pick-and-place without requiring task-specific training.

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction, and Planning

Overview

The paper "V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning" introduces a framework for developing self-supervised video models capable of understanding, predicting, and planning actions in the physical world. The authors propose V-JEPA 2, an action-free joint-embedding-predictive architecture that leverages over 1 million hours of video data for pre-training. This approach achieves state-of-the-art performance across various tasks, including motion understanding, human action anticipation, and video question-answering, by aligning V-JEPA 2 with a LLM.

Figure 1: V-JEPA 2 Overview.

Pre-training Methodology

The pre-training of V-JEPA 2 involves utilizing video data to learn representations that capture the dynamics of the world. The video model is pre-trained using a mask denoising objective that predicts missing segments in a learned representation space. This joint-embedding predictive architecture enables V-JEPA 2 to perform well in understanding and prediction tasks.

Figure 2: Multistage training.

The pre-training exploits large-scale internet video data without direct interaction data, setting a basis for subsequent stages. The architecture comprises an encoder and a predictor, both of which are vision transformers. A key aspect of scaling involves extending the video duration and resolution used during training, enabling the model to learn more detailed representations that improve downstream task performance.

Figure 3: Data Scaling {additional_guidance} Curation.

Action-Conditioned Post-Training

Post-training involves refining V-JEPA 2 into an action-conditioned world model, V-JEPA 2-AC, using a smaller dataset of interaction data from robotic trajectories. This process incorporates an autoregressive feature prediction objective, allowing the model to anticipate future states conditioned on actions.

Figure 4: V-JEPA 2-AC training.

This stage leverages the previously learned representations, focusing on the causal effect of actions. During inference, planning is executed by minimizing a goal-conditioned energy function, facilitating zero-shot deployment on robots for tasks like object picking and placing.

Zero-shot Planning and Control

V-JEPA 2-AC demonstrates its ability to perform prehensile manipulation tasks in a zero-shot manner on robotic systems. The model achieves significant success in tasks such as grasping and pick-and-place through visual goal conditioning, without requiring task-specific training or reward shaping.

Figure 5: Planning.

Figure 6: Pick-{additional_guidance}-Place.

Evaluation and Results

The paper showcases V-JEPA 2's performance on several benchmarks illustrating motion and appearance understanding, action anticipation, and video question-answering:

• Understanding: The model achieves high accuracy across visual tasks, indicating strong generalization capabilities.
• Prediction: With a new action anticipation benchmark, V-JEPA 2 improves recognition performance by a significant margin compared to existing models.
• Planning: V-JEPA 2-AC enables effective zero-shot planning, demonstrating potential for practical real-world deployment.

Figure 7: Single-Goal Reaching.

Implications and Future Work

The introduction of a scalable, self-supervised framework for video modeling opens up avenues for developing autonomous systems with robust perceptual and planning capabilities. Future developments could focus on extending V-JEPA 2 to accommodate longer planning horizons and enhance its ability to work with diverse types of data and environments.

Conclusion

The V-JEPA 2 framework capitalizes on large-scale self-supervised learning from video data to produce models that excel in understanding, prediction, and planning tasks. By integrating video data with robot interaction data, this self-supervised approach achieves impressive results, positioning it as a viable method for developing world models that autonomously operate in complex environments.