Papers
Topics
Authors
Recent
Search
2000 character limit reached

Language to Rewards for Robotic Skill Synthesis

Published 14 Jun 2023 in cs.RO, cs.AI, and cs.LG | (2306.08647v2)

Abstract: LLMs have demonstrated exciting progress in acquiring diverse new capabilities through in-context learning, ranging from logical reasoning to code-writing. Robotics researchers have also explored using LLMs to advance the capabilities of robotic control. However, since low-level robot actions are hardware-dependent and underrepresented in LLM training corpora, existing efforts in applying LLMs to robotics have largely treated LLMs as semantic planners or relied on human-engineered control primitives to interface with the robot. On the other hand, reward functions are shown to be flexible representations that can be optimized for control policies to achieve diverse tasks, while their semantic richness makes them suitable to be specified by LLMs. In this work, we introduce a new paradigm that harnesses this realization by utilizing LLMs to define reward parameters that can be optimized and accomplish variety of robotic tasks. Using reward as the intermediate interface generated by LLMs, we can effectively bridge the gap between high-level language instructions or corrections to low-level robot actions. Meanwhile, combining this with a real-time optimizer, MuJoCo MPC, empowers an interactive behavior creation experience where users can immediately observe the results and provide feedback to the system. To systematically evaluate the performance of our proposed method, we designed a total of 17 tasks for a simulated quadruped robot and a dexterous manipulator robot. We demonstrate that our proposed method reliably tackles 90% of the designed tasks, while a baseline using primitive skills as the interface with Code-as-policies achieves 50% of the tasks. We further validated our method on a real robot arm where complex manipulation skills such as non-prehensile pushing emerge through our interactive system.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (55)
  1. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  2. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  3. Code as policies: Language model programs for embodied control. arXiv preprint arXiv:2209.07753, 2022.
  4. Socratic models: Composing zero-shot multimodal reasoning with language. arXiv preprint arXiv:2204.00598, 2022.
  5. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.
  6. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
  7. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022.
  8. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021.
  9. Do as i can, not as i say: Grounding language in robotic affordances. arXiv preprint arXiv:2204.01691, 2022.
  10. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
  11. Context-aware language modeling for goal-oriented dialogue systems. arXiv preprint arXiv:2204.10198, 2022.
  12. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. In International Conference on Machine Learning, pages 9118–9147. PMLR, 2022.
  13. Progprompt: Generating situated robot task plans using large language models. arXiv preprint arXiv:2209.11302, 2022.
  14. BC-z: Zero-shot task generalization with robotic imitation learning. In 5th Annual Conference on Robot Learning, 2021. URL https://openreview.net/forum?id=8kbp23tSGYv.
  15. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
  16. Interactive language: Talking to robots in real time. arXiv preprint arXiv:2210.06407, 2022.
  17. Robust recovery controller for a quadrupedal robot using deep reinforcement learning. arXiv preprint arXiv:1901.07517, 2019.
  18. Sim-to-real learning of all common bipedal gaits via periodic reward composition. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 7309–7315. IEEE, 2021.
  19. Relmogen: Leveraging motion generation in reinforcement learning for mobile manipulation. arXiv preprint arXiv:2008.07792, 2020.
  20. Learning navigation behaviors end-to-end with autorl. IEEE Robotics and Automation Letters, 4(2):2007–2014, 2019.
  21. Predictive Sampling: Real-time Behaviour Synthesis with MuJoCo. dec 2022. doi:10.48550/arXiv.2212.00541. URL https://arxiv.org/abs/2212.00541.
  22. Using natural language for reward shaping in reinforcement learning. arXiv preprint arXiv:1903.02020, 2019.
  23. Inferring rewards from language in context. arXiv preprint arXiv:2204.02515, 2022.
  24. Minedojo: Building open-ended embodied agents with internet-scale knowledge. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https://openreview.net/forum?id=rc8o_j8I8PX.
  25. Correcting robot plans with natural language feedback. In Robotics: Science and Systems (RSS), 2022.
  26. Reward design with language models. In International Conference on Learning Representations (ICLR), 2023.
  27. H. Hu and D. Sadigh. Language instructed reinforcement learning for human-ai coordination. In 40th International Conference on Machine Learning (ICML), 2023.
  28. Translating structured english to robot controllers. Advanced Robotics, 22(12):1343–1359, 2008.
  29. Learning to parse natural language commands to a robot control system. In Experimental robotics: the 13th international symposium on experimental robotics, pages 403–415. Springer, 2013.
  30. Room-across-room: Multilingual vision-and-language navigation with dense spatiotemporal grounding. arXiv preprint arXiv:2010.07954, 2020.
  31. A new path: Scaling vision-and-language navigation with synthetic instructions and imitation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10813–10823, 2023.
  32. Grounding language with visual affordances over unstructured data. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), London, UK, 2023.
  33. Bridge data: Boosting generalization of robotic skills with cross-domain datasets. arXiv preprint arXiv:2109.13396, 2021.
  34. From language to goals: Inverse reinforcement learning for vision-based instruction following. arXiv preprint arXiv:1902.07742, 2019.
  35. Lila: Language-informed latent actions. In Proceedings of the 5th Conference on Robot Learning (CoRL), 2021.
  36. Program synthesis with large language models. arXiv:2108.07732, 2021.
  37. Dreamcoder: Growing generalizable, interpretable knowledge with wake-sleep bayesian program learning. arXiv:2006.08381, 2020.
  38. Competition-level code generation with alphacode. Science, 378(6624):1092–1097, 2022.
  39. A large-scale benchmark for few-shot program induction and synthesis. In ICML, 2021.
  40. Learning abstract structure for drawing by efficient motor program induction. NeurIPS, 2020.
  41. Learning to synthesize programs as interpretable and generalizable policies. NeurIPS, 2021.
  42. Learning language-conditioned robot behavior from offline data and crowd-sourced annotation. In A. Faust, D. Hsu, and G. Neumann, editors, Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 1303–1315. PMLR, 08–11 Nov 2022. URL https://proceedings.mlr.press/v164/nair22a.html.
  43. Learning to understand goal specifications by modelling reward. arXiv preprint arXiv:1806.01946, 2018.
  44. Real-time natural language corrections for assistive robotic manipulators. International Journal of Robotics Research (IJRR), 36:684–698, 2017.
  45. “no, to the right”–online language corrections for robotic manipulation via shared autonomy. arXiv preprint arXiv:2301.02555, 2023.
  46. Reshaping robot trajectories using natural language commands: A study of multi-modal data alignment using transformers. In International Conference on Intelligent Robots and Systems (IROS), pages 978–984, 2022a.
  47. Latte: Language trajectory transformer. arXiv preprint arXiv:2208.02918, 2022b.
  48. Large language models are built-in autoregressive search engines. arXiv preprint arXiv:2305.09612, 2023.
  49. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026–5033. IEEE, 2012. doi:10.1109/IROS.2012.6386109.
  50. OpenAI. Gpt-4 technical report. arXiv, 2023.
  51. Learning and adapting agile locomotion skills by transferring experience. arXiv preprint arXiv:2304.09834, 2023.
  52. Opt-mimic: Imitation of optimized trajectories for dynamic quadruped behaviors. arXiv preprint arXiv:2210.01247, 2022.
  53. Barkour: Benchmarking animal-level agility with quadruped robots. arXiv preprint arXiv:2305.14654, 2023.
  54. F-vlm: Open-vocabulary object detection upon frozen vision and language models. arXiv preprint arXiv:2209.15639, 2023.
  55. E. Olson. Apriltag: A robust and flexible visual fiducial system. 2023.
Citations (213)
View on Semantic Scholar

Summary

  • The paper introduces a novel LLM-based framework converting natural language into optimized reward functions.
  • It employs a dual system of Reward Translator and Motion Controller using MuJoCo MPC for real-time optimization.
  • Experiments on 17 tasks yield a 90% success rate, demonstrating significant improvement over primitive skill methods.

Language to Rewards for Robotic Skill Synthesis: An Overview

The research paper titled "Language to Rewards for Robotic Skill Synthesis" introduces an innovative method for applying LLMs in the context of robotic control. The key insight of this work lies in leveraging LLMs to interface natural language instructions with reward parameters that can be optimized for robotic tasks, instead of attempting to output low-level robot commands directly, which are often hardware-specific and underrepresented in LLM training data. This approach harnesses the semantic richness of reward functions, contributing to the field of robotics by providing a flexible and efficient paradigm for skill synthesis.

Methodology

The authors propose a system composed of two main components: the Reward Translator and the Motion Controller. The Reward Translator, based on LLMs, interprets user instructions to generate reward specifications. This is achieved in two stages: First, a Motion Descriptor LLM converts the input into a detailed natural language description of the robot motion. Second, a Reward Coder LLM translates this description into reward parameters that guide the Motion Controller.

For the Motion Controller, the authors employ MuJoCo MPC, a model predictive control tool that optimizes the generated reward functions in real-time. This optimization facilitates interactive robot behavior synthesis, allowing users to provide feedback and corrections.

Experimental Validation

The research evaluates the proposed method across 17 tasks using a simulated quadruped robot and a dexterous manipulator robot. The tasks range from basic locomotion and manipulation to more complex skills. The method demonstrates a significant success rate, reliably achieving 90% of the tasks, compared to 50% with a baseline method using primitive skills as an interface. Notably, the approach shows strong capability in solving new tasks with minimal pre-engineered control primitives.

Findings and Implications

The study's results underscore the potential of using reward functions as an interface for mapping language to robotic actions. This approach offers several advantages:

  1. Expressiveness and Flexibility: By generating reward functions, the system is not limited to pre-defined, low-level primitives, allowing for the synthesis of novel and complex behaviors.
  2. Interactivity: The real-time optimization and user feedback loop empower users to iteratively refine robot actions, making the system both adaptable and user-friendly.
  3. Reduced Engineering Effort: The LLM-driven reward specification minimizes the need for expert-designed control strategies, highlighting a path towards more accessible robotic programming.

Future Directions

The paper suggests several future research avenues. First, integrating multi-modal inputs beyond natural language could enhance the expressive power of the system. Additionally, automating or generalizing the motion description templates for application to new robot morphologies would increase the method's portability. Finally, encompassing dynamic, time-varying rewards could open up new task domains and complexity levels.

In conclusion, this paper presents a compelling approach to robotic skill acquisition through the lens of LLMs and reward-based optimization. By establishing a robust link between language and action via reward parameters, it paves the way for advanced robotic systems capable of interpreting and executing complex human instructions with reduced dependency on extensive data or specialized knowledge. The potential impact of such systems stretches across numerous domains, from automated industrial processes to personalized service robotics.

Paper to Video (Beta)

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up