TidyBot: Personalized Robot Assistance with Large Language Models

Abstract: For a robot to personalize physical assistance effectively, it must learn user preferences that can be generally reapplied to future scenarios. In this work, we investigate personalization of household cleanup with robots that can tidy up rooms by picking up objects and putting them away. A key challenge is determining the proper place to put each object, as people's preferences can vary greatly depending on personal taste or cultural background. For instance, one person may prefer storing shirts in the drawer, while another may prefer them on the shelf. We aim to build systems that can learn such preferences from just a handful of examples via prior interactions with a particular person. We show that robots can combine language-based planning and perception with the few-shot summarization capabilities of LLMs to infer generalized user preferences that are broadly applicable to future interactions. This approach enables fast adaptation and achieves 91.2% accuracy on unseen objects in our benchmark dataset. We also demonstrate our approach on a real-world mobile manipulator called TidyBot, which successfully puts away 85.0% of objects in real-world test scenarios.

• The paper introduces a novel method that leverages large language models to infer personalized object placement rules from minimal user input.
• It employs few-shot learning and text summarization to bypass extensive data collection and manual labeling in robotic applications.
• Practical tests on the TidyBot platform show 91.2% accuracy for unseen objects in the lab and 85% correct placement in real-world settings.

Personalized Robot Assistance with LLMs: TidyBot

The paper "TidyBot: Personalized Robot Assistance with LLMs" introduces an innovative approach to personalizing robotic assistance for household tasks, specifically focused on cleaning and organizing tasks like tidying up a room. The highlighted challenge in this domain is the personalization of object placement preferences, which can vary remarkably based on individual tastes or cultural norms. Unlike traditional methods that require extensive user-specific data collection, this work leverages the generalization capabilities of LLMs to infer user preferences from minimal inputs.

The methodology employs LLMs in a novel manner, utilizing their few-shot learning and summarization capabilities to extrapolate user preferences from a handful of examples. By integrating language-based planning and perception, the system can establish generalized rules for object placement that align with personal preferences and are applicable in future interactions with the user. The reported accuracy of this system in laboratory benchmarks, reaching 91.2% for unseen objects, underscores its potential for practical household scenarios.

Central to this approach is the use of LLMs to synthesize generalizable sorting rules from user-provided examples, expressed in a simple textual input format. This method circumvents the need for extensive dataset construction and costly manual labeling, directly coupling the semantic understanding present within LLMs to the practical utility in robotics. The paper also provides an in-depth evaluation using a real-world robotic platform named TidyBot, achieving an 85% success rate in placing objects correctly during physical tests.

This work presents several contributions to service robotics. First, it advances the concept that LLM text summarization serves as a robust mechanism for generalization, bridging the gap between few-shot learning paradigms and practical robotic applications. Secondly, it introduces a new benchmark dataset for evaluating the generalization capability in receptacle selection, enhancing reproducibility and facilitating future research comparisons. Lastly, it demonstrates the implementation of this approach on a functional mobile manipulation system, showcasing its feasibility outside simulated environments.

The potential implications for future AI developments are significant. By reducing the barrier to entry for personalizing robotic systems, this method accelerates progress toward more adaptable and user-centric robotic solutions. Practically, the approach could be extended to various contexts within robotics, influencing areas such as autonomous sorting, custom logistics, and personalized home assistance.

For future research, this paper opens avenues for exploring more complex interaction models between robotics and users, potentially incorporating feedback mechanisms to refine LLM outputs. Moreover, continued advancements in visual LLMs could further enhance object classification and perception accuracy, addressing limitations observed in the evaluation. In addition to improving summarization accuracy, research could aim to enhance the flexibility of manipulation primitives, thereby expanding the functional domain of TidyBot-like systems in home environments.

In summation, the presented research exemplifies a forward-thinking approach to personalizing robotic assistance, marking a significant step toward more intuitive and intelligent home robotics systems.