Papers
Topics
Authors
Recent
Search
2000 character limit reached

Never-Ending Behavior-Cloning Agent for Robotic Manipulation

Published 1 Mar 2024 in cs.RO and cs.AI | (2403.00336v2)

Abstract: Relying on multi-modal observations, embodied robots could perform multiple robotic manipulation tasks in unstructured real-world environments. However, most language-conditioned behavior-cloning agents still face existing long-standing challenges, i.e., 3D scene representation and human-level task learning, when adapting into new sequential tasks in practical scenarios. We here investigate these above challenges with NBAgent in embodied robots, a pioneering language-conditioned Never-ending Behavior-cloning Agent. It can continually learn observation knowledge of novel 3D scene semantics and robot manipulation skills from skill-shared and skill-specific attributes, respectively. Specifically, we propose a skill-sharedsemantic rendering module and a skill-shared representation distillation module to effectively learn 3D scene semantics from skill-shared attribute, further tackling 3D scene representation overlooking. Meanwhile, we establish a skill-specific evolving planner to perform manipulation knowledge decoupling, which can continually embed novel skill-specific knowledge like human from latent and low-rank space. Finally, we design a never-ending embodied robot manipulation benchmark, and expensive experiments demonstrate the significant performance of our method. Visual results, code, and dataset are provided at: https://neragent.github.io.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (52)
  1. Few-shot continual active learning by a robot. In NeurIPS, volume 35, pp.  30612–30624, 2022.
  2. Cbcl-pr: A cognitively inspired model for class-incremental learning in robotics. IEEE Transactions on Cognitive and Developmental Systems, 2023.
  3. Continual learning through human-robot interaction–human perceptions of a continual learning robot in repeated interactions. arXiv preprint arXiv:2305.16332, 2023.
  4. Rainbow memory: Continual learning with a memory of diverse samples. In CVPR, pp.  8218–8227, 2021.
  5. Rt-1: Robotics transformer for real-world control at scale. arXiv preprint arXiv:2212.06817, 2022.
  6. Do as i can, not as i say: Grounding language in robotic affordances. In CoRL, pp.  287–318. PMLR, 2023.
  7. On tiny episodic memories in continual learning. arXiv preprint arXiv:1902.10486, 2019.
  8. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113, 2023.
  9. Kernel continual learning. In ICML, pp.  2621–2631. PMLR, 2021.
  10. Heterogeneous forgetting compensation for class-incremental learning. In ICCV, pp.  11742–11751, 2023.
  11. Podnet: Pooled outputs distillation for small-tasks incremental learning. In ECCV, pp.  86–102. Springer, 2020.
  12. Reinforcement learning with neural radiance fields. In NeurIPS, volume 35, pp.  16931–16945, 2022.
  13. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.03378, 2023.
  14. Cril: Continual robot imitation learning via generative and prediction model. In IROS, pp.  6747–5754. IEEE, 2021.
  15. Ifor: Iterative flow minimization for robotic object rearrangement. In CVPR, pp.  14787–14797, 2022.
  16. Rvt: Robotic view transformer for 3d object manipulation. arXiv preprint arXiv:2306.14896, 2023.
  17. Continual robot learning using self-supervised task inference. IEEE Transactions on Cognitive and Developmental Systems, 2023.
  18. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
  19. Inner monologue: Embodied reasoning through planning with language models. arXiv preprint arXiv:2207.05608, 2022.
  20. Perceiver io: A general architecture for structured inputs & outputs. arXiv preprint arXiv:2107.14795, 2021.
  21. Rlbench: The robot learning benchmark & learning environment. IEEE Robotics and Automation Letters, 5(2):3019–3026, 2020.
  22. Coarse-to-fine q-attention: Efficient learning for visual robotic manipulation via discretisation. In CVPR, pp.  13739–13748, 2022.
  23. Vima: General robot manipulation with multimodal prompts. arXiv, 2022.
  24. Continual learning with node-importance based adaptive group sparse regularization. In NeurIPS, volume 33, pp.  3647–3658, 2020.
  25. Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), ICML, pp.  1207–1216, Stanford, CA, 2000. Morgan Kaufmann.
  26. Your diffusion model is secretly a zero-shot classifier. In ICCV, pp.  2206–2217, October 2023.
  27. Continual few-shot intent detection. In COLING, pp.  333–343, 2022.
  28. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017.
  29. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
  30. Learning transferable visual models from natural language supervision. In ICML, pp.  8748–8763. PMLR, 2021.
  31. icarl: Incremental classifier and representation learning. In CVPR, pp.  2001–2010, 2017.
  32. V-rep: A versatile and scalable robot simulation framework. In IROS, pp.  1321–1326. IEEE, 2013.
  33. High-resolution image synthesis with latent diffusion models. In CVPR, pp.  10684–10695, 2022.
  34. Lm-nav: Robotic navigation with large pre-trained models of language, vision, and action. In CoRL, pp.  492–504. PMLR, 2023.
  35. Snerl: Semantic-aware neural radiance fields for reinforcement learning. In ICML. PMLR, 2023.
  36. Cliport: What and where pathways for robotic manipulation. In CoRL, pp.  894–906. PMLR, 2022.
  37. Perceiver-actor: A multi-task transformer for robotic manipulation. In CoRL, pp.  785–799. PMLR, 2023.
  38. Create your world: Lifelong text-to-image diffusion. arXiv preprint arXiv:2309.04430, 2023.
  39. Exploring example influence in continual learning. In NeurIPS, volume 35, pp.  27075–27086, 2022.
  40. Gcr: Gradient coreset based replay buffer selection for continual learning. In CVPR, pp.  99–108, 2022.
  41. Bring evanescent representations to life in lifelong class incremental learning. In CVPR, pp.  16732–16741, 2022.
  42. Lotus: Continual imitation learning for robot manipulation through unsupervised skill discovery. arXiv preprint arXiv:2311.02058, 2023.
  43. Coscl: Cooperation of small continual learners is stronger than a big one. In ECCV, pp.  254–271. Springer, 2022.
  44. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint arXiv:2302.00487, 2023.
  45. Incremental learning via rate reduction. In CVPR, pp.  1125–1133, 2021.
  46. Incremental learning using conditional adversarial networks. In ICCV, pp.  6619–6628, 2019.
  47. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, pp.  2955–2966, 2023.
  48. Continual object detection via prototypical task correlation guided gating mechanism. In CVPR, pp.  9255–9264, 2022.
  49. Large batch optimization for deep learning: Training bert in 76 minutes. arXiv preprint arXiv:1904.00962, 2019.
  50. pixelnerf: Neural radiance fields from one or few images. In CVPR, pp.  4578–4587, 2021.
  51. Gnfactor: Multi-task real robot learning with generalizable neural feature fields. In CoRL, pp.  284–301. PMLR, 2023.
  52. Rt-2: Vision-language-action models transfer web knowledge to robotic control. In CoRL, pp.  2165–2183. PMLR, 2023.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 0 likes about this paper.

Sign Up for Free