Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery

Abstract: A general-purpose service robot (GPSR), which can execute diverse tasks in various environments, requires a system with high generalizability and adaptability to tasks and environments. In this paper, we first developed a top-level GPSR system for worldwide competition (RoboCup@Home 2023) based on multiple foundation models. This system is both generalizable to variations and adaptive by prompting each model. Then, by analyzing the performance of the developed system, we found three types of failure in more realistic GPSR application settings: insufficient information, incorrect plan generation, and plan execution failure. We then propose the self-recovery prompting pipeline, which explores the necessary information and modifies its prompts to recover from failure. We experimentally confirm that the system with the self-recovery mechanism can accomplish tasks by resolving various failure cases. Supplementary videos are available at https://sites.google.com/view/srgpsr .

• The paper introduces a novel GPSR system that integrates foundation models with prompt-based self-recovery for dynamic task execution.
• It addresses common failure modes, including insufficient information, planning errors, and execution issues in real-world settings.
• Experimental results from RoboCup@Home 2023 demonstrate the system’s adaptability and robustness in complex service scenarios.

An Overview of "Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery"

The paper under review, "Self-Recovery Prompting: Promptable General Purpose Service Robot System with Foundation Models and Self-Recovery," presents a comprehensive study on the development and deployment of a General-Purpose Service Robot (GPSR) system. This research, conducted by a team from the University of Tokyo, explores the intricacies of using foundation models to enhance the adaptability and generalizability of service robots to execute diverse tasks in dynamic environments.

Summary of the Proposed System

The central contribution of the paper lies in the development of a novel GPSR system that integrates multiple foundation models. These models, namely Whisper, GPT-4, Detic, and CLIP, are utilized to handle various robotic functionalities such as speech recognition, task planning, and object recognition. This system is designed to compete in challenging settings such as the RoboCup@Home 2023, with success denoted by its first and second placements in domestic and worldwide competitions, respectively.

The system's adaptability stems from its reliance on prompt-based responses, which allows for performance tuning without retraining. The researchers demonstrate the system's robustness in handling competitive environments and highlight its ability to process commands given in natural language.

Identification of Failure Modes

A significant finding from the study is the identification of three common failure modes encountered in realistic settings that GPSR systems must navigate:

1. Insufficient Information: Occurs when tasks require data that the system does not have pre-registered, impeding task execution.
2. Incorrect Plan Generation: This arises from planning errors due to misinterpretation or ambiguous command inputs, affecting the sequence of actions generated by the system.
3. Plan Execution Failure: These failures are predominantly due to the physical or operational constraints in real-world environments that prevent successful task completion.

Self-Recovery Mechanism

In response to these identified failure modes, the authors propose a self-recovery prompting pipeline. This mechanism allows the GPSR system to dynamically adjust and modify its inputs and plans by leveraging foundation models to "learn" from previous experiences and human interactions. Notably, the system can conduct re-planning in light of execution failures or request additional information from human operators to fill missing data gaps, hence enhancing its operational robustness.

Experimental Validation

The system's effectiveness is validated through experiments involving handcrafted tasks designed to evaluate the system's ability to navigate the discussed failure modes. The authors report successful task completions across diverse scenarios, demonstrating the practical utility of the self-recovery mechanism. The experiments confirm the capability of the GPSR system to adaptively address failures and optimize task performance through interactive recovery procedures.

Implications and Future Directions

From a practical standpoint, the introduction of a self-recovery mechanism significantly strengthens the resilience and adaptability of service robots operating in complex, uncontrolled environments. Theoretically, it highlights the potential for foundation models to enhance robotic intelligence not only by pre-training but through interactive prompting and real-time adaptation.

Future research avenues could focus on expanding the variety of foundation models integrated into the system, exploring more sophisticated interactive techniques for handling failures, and developing comprehensive benchmarks for evaluating the performance of GPSR systems in diverse operational contexts.

In conclusion, this paper contributes substantially to the robotics field by demonstrating the efficacy of foundation models and self-recovery prompting in developing robust and efficient general-purpose service robots.