Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models

Abstract: A central challenge towards developing robots that can relate human language to their perception and actions is the scarcity of natural language annotations in diverse robot datasets. Moreover, robot policies that follow natural language instructions are typically trained on either templated language or expensive human-labeled instructions, hindering their scalability. To this end, we introduce NILS: Natural language Instruction Labeling for Scalability. NILS automatically labels uncurated, long-horizon robot data at scale in a zero-shot manner without any human intervention. NILS combines pretrained vision-language foundation models in order to detect objects in a scene, detect object-centric changes, segment tasks from large datasets of unlabelled interaction data and ultimately label behavior datasets. Evaluations on BridgeV2, Fractal, and a kitchen play dataset show that NILS can autonomously annotate diverse robot demonstrations of unlabeled and unstructured datasets while alleviating several shortcomings of crowdsourced human annotations, such as low data quality and diversity. We use NILS to label over 115k trajectories obtained from over 430 hours of robot data. We open-source our auto-labeling code and generated annotations on our website: http://robottasklabeling.github.io.

• The paper introduces NILS, a zero-shot labeling framework that scales robot policy learning by automatically annotating over 115,000 trajectories without human intervention.
• It employs a three-stage process using vision-language models to detect objects, label object-centric actions, and identify key task states.
• Evaluations on datasets like BridgeV2, Fractal, and a kitchen play dataset show that NILS outperforms current state-of-the-art models in task segmentation and annotation detail.

Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models: An Expert Overview

The paper "Scaling Robot Policy Learning via Zero-Shot Labeling with Foundation Models" introduces a novel framework named Natural Language Instruction Labeling for Scalability (NILS) for labeling robot demonstrations. The primary objective of NILS is to address the limitations posed by the scarcity of natural language annotations in existing robot datasets, which is essential for developing robots that can easily relate human language to their perception and actions. Typically, language-conditioned robot policies are trained using templated annotations or expensive human-generated instructions, limiting their scalability. NILS offers a zero-shot labeling solution that operates without human intervention, enabling it to efficiently annotate large-scale, uncurated, long-horizon robot datasets.

At the core of NILS is its capability to leverage pretrained vision-language foundation models to identify objects, detect object-centric changes, segment tasks, and label behavior datasets autonomously. The evaluation of NILS was conducted on various datasets, including BridgeV2, Fractal, and a kitchen play dataset, demonstrating its capability to effectively annotate large volumes of unlabeled robot data, totaling over 115,000 trajectories and 430 hours of robot data.

A significant contribution of NILS is its ability to provide high-quality annotations while mitigating issues associated with crowdsourced human-generated labels such as inconsistencies and limited diversity. The paper shows that NILS surpasses existing state-of-the-art video-LLMs in identifying keystates and annotates tasks with a greater level of detail.

Methodology

The NILS framework is divided into three stages:

1. Stage 1 concerns the identification of objects in a scene, utilizing multiple vision-LLMs to detect and consistently name objects despite occlusions.
2. Stage 2 focuses on object-centric scene annotation, monitoring four key signals: object relations and movement, object state changes, gripper position, and gripper closing actions.
3. Stage 3 involves the detection of keystates and the generation of language labels, employing a heuristic consensus approach to identify important keystates and LLMs to approximate free-form instructions.

Implications and Future Directions

The implications of NILS are profound in both practical and theoretical domains. Practically, the ability to automatically label large datasets significantly reduces the cost and labor associated with training language-conditioned robot policies. Theoretically, it advances the notion of scalable robot learning by demonstrating the efficacy of foundation models in generating detailed and contextually relevant annotations.

The results of employing NILS indicate a promising direction for future developments in artificial intelligence, particularly in the domain of robotics. The framework's application of multiple specialist models highlights the potential for modular approaches to improve robotic perception and interaction capabilities. Additionally, the insights gained from this work may stimulate further research into refining foundation models for more specialized tasks and environments.

Overall, NILS exemplifies a forward-looking approach to overcoming the challenges of robot policy learning, laying groundwork for more intuitive and capable robotic systems that can better navigate and interact with the complexities of the real world. As foundation models continue to evolve, frameworks like NILS will undoubtedly catalyze further breakthroughs in scalable, autonomous machine learning systems.