Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets

Abstract: The pre-training of visual representations has enhanced the efficiency of robot learning. Due to the lack of large-scale in-domain robotic datasets, prior works utilize in-the-wild human videos to pre-train robotic visual representation. Despite their promising results, representations from human videos are inevitably subject to distribution shifts and lack the dynamics information crucial for task completion. We first evaluate various pre-trained representations in terms of their correlation to the downstream robotic manipulation tasks (i.e., manipulation centricity). Interestingly, we find that the "manipulation centricity" is a strong indicator of success rates when applied to downstream tasks. Drawing from these findings, we propose Manipulation Centric Representation (MCR), a foundation representation learning framework capturing both visual features and the dynamics information such as actions and proprioceptions of manipulation tasks to improve manipulation centricity. Specifically, we pre-train a visual encoder on the DROID robotic dataset and leverage motion-relevant data such as robot proprioceptive states and actions. We introduce a novel contrastive loss that aligns visual observations with the robot's proprioceptive state-action dynamics, combined with a behavior cloning (BC)-like actor loss to predict actions during pre-training, along with a time contrastive loss. Empirical results across 4 simulation domains with 20 tasks verify that MCR outperforms the strongest baseline method by 14.8%. Moreover, MCR boosts the performance of data-efficient learning with a UR5e arm on 3 real-world tasks by 76.9%. Project website: https://robots-pretrain-robots.github.io/.

• The paper introduces MCR, a novel representation learning framework that improves robotic manipulation performance by leveraging dynamics labels and large-scale datasets.
• The study demonstrates that aligning visual observations with proprioceptive dynamics via contrastive loss leads to a 14.8% increase in simulation success rates.
• MCR is validated through extensive evaluations on both simulated and real-world tasks, highlighting its robustness and practical applicability in robotics.

Manipulation-Centric Robotic Representation Learning

The paper "Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets" introduces a new framework called Manipulation Centric Representation (MCR) to enhance the training of robotic visual representations. The study underscores the importance of "manipulation centricity," a metric that correlates strongly with downstream robotic task performance, and explores the use of large-scale robotic datasets over human datasets for improved representation learning.

Introduction to Manipulation Centricity and MCR

Manipulation centricity is defined as the representation's capability to highlight manipulation-relevant regions like robot end-effectors and task-specific objects. The authors propose MCR, a framework that leverages dynamics labels within a robotics dataset, contrasting with previous methods that used human-centric data, which lacked task-specific dynamics and introduced substantial embodiment gaps.

Figure 1: The proposed MCR framework learns manipulation-centric representations from large-scale datasets, validated by enhanced downstream policy performances.

The method employs a contrastive loss that aligns visual observations with robot proprioceptive dynamics and an additional loss for action prediction, enhancing the representation's temporal and task-centric fidelity.

Experimental Setup and Evaluation

The authors adopt a cross-validation strategy to evaluate their framework across both simulated and real-world robotic tasks. Pre-trained representations are assessed using Im. Learning (IL) techniques, with the policy performance indicating representation quality.

Figure 2: Strong correlation observed between manipulation centricity and downstream task performance, highlighting MCR's efficacy.

Simulation Tasks

The evaluation encompasses 20 complex tasks within four simulation domains, focusing on diversity in robot types and task complexity.

Figure 3: Tasks include a variety of manipulation scenarios across several domains, serving as benchmarks to showcase MCR's robustness.

The results demonstrate that MCR achieves superior task performance relative to baseline models, exhibiting a 14.8% increase in success rates across simulations.

Real-World Testing

The real-world experiments, conducted with a UR5e robot setup, further validate MCR's effectiveness. MCR significantly outperformed baselines, showcasing substantial gains in task success rates, demonstrating the method's practical applicability.

Figure 4: Real robot tasks designed to test MCR's generalizability across various manipulation skills.

Implementation and Performance

MCR is implemented using a ResNet-50 backbone, trained extensively with detailed attention to temporal dynamics and action relevance. The training employs a large-scale dataset (DROID) comprising extensive robot trajectory data, emphasizing dynamics information absent in prior studies.

Figure 5: MCR's Grad-CAM visualization reveals optimal manipulation-centric attention compared to baselines.

The efficiency of the MCR training process, alongside its performance improvements, indicates significant strides in pre-trained robotic representations. The use of large-scale robot datasets like DROID facilitates this progress, reducing domain-related generalization issues.

Conclusion and Future Directions

MCR provides a novel manipulation-centric lens to guide robotic representation learning, substantially improving task performance in both simulated and real-world scenarios. Future work could explore integrating language and multi-modal inputs to further enhance the representation's task specificity and temporal coherence.

The approach paves the way for enhanced data utilization strategies, emphasizing the importance of dynamics information and pre-training efficiency in robotic learning paradigms. As large-scale robotic datasets become more prevalent, frameworks like MCR could serve as benchmarks for future AI systems in embodied learning environments.