Teaching Robots Where To Go And How To Act With Human Sketches via Spatial Diagrammatic Instructions
Abstract: This paper introduces Spatial Diagrammatic Instructions (SDIs), an approach for human operators to specify objectives and constraints that are related to spatial regions in the working environment. Human operators are enabled to sketch out regions directly on camera images that correspond to the objectives and constraints. These sketches are projected to 3D spatial coordinates, and continuous Spatial Instruction Maps (SIMs) are learned upon them. These maps can then be integrated into optimization problems for tasks of robots. In particular, we demonstrate how Spatial Diagrammatic Instructions can be applied to solve the Base Placement Problem of mobile manipulators, which concerns the best place to put the manipulator to facilitate a certain task. Human operators can specify, via sketch, spatial regions of interest for a manipulation task and permissible regions for the mobile manipulator to be at. Then, an optimization problem that maximizes the manipulator's reachability, or coverage, over the designated regions of interest while remaining in the permissible regions is solved. We provide extensive empirical evaluations, and show that our formulation of Spatial Instruction Maps provides accurate representations of user-specified diagrammatic instructions. Furthermore, we demonstrate that our diagrammatic approach to the Mobile Base Placement Problem enables higher quality solutions and faster runtime.
- H. Ravichandar, A. S. Polydoros, S. Chernova, and A. Billard, “Recent advances in robot learning from demonstration,” Annual review of control, robotics, and autonomous systems, 2020.
- A. Paraschos, C. Daniel, J. Peters, and G. Neumann, “Probabilistic movement primitives,” in Proceedings of the 26th International Conference on Neural Information Processing Systems, 2013.
- J. Kober and J. Peters, “Learning motor primitives for robotics,” in IEEE International Conference on Robotics and Automation, 2009.
- W. Zhi, T. Lai, L. Ott, and F. Ramos, “Diffeomorphic transforms for generalised imitation learning,” in Learning for Dynamics and Control Conference, L4DC, 2022.
- M. A. Rana, M. Mukadam, S. R. Ahmadzadeh, S. Chernova, and B. Boots, “Towards robust skill generalization: Unifying learning from demonstration and motion planning,” in Proceedings of the 1st Annual Conference on Robot Learning, 2017.
- S. Griffith, K. Subramanian, J. Scholz, C. L. Isbell, and A. L. Thomaz, “Policy shaping: Integrating human feedback with reinforcement learning,” in Advances in Neural Information Processing Systems, C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Weinberger, Eds., vol. 26. Curran Associates, Inc., 2013.
- S. Tellex, N. Gopalan, H. Kress-Gazit, and C. Matuszek, “Robots that use language,” Annu. Rev. Control. Robotics Auton. Syst., 2020.
- W. Zhi, T. Zhang, and M. Johnson-Roberson, “Learning from demonstration via probabilistic diagrammatic teaching,” in IEEE International Conference on Robotics and Automation (ICRA), 2024.
- A. Elfes, “Sonar-based real-world mapping and navigation,” IEEE Journal on Robotics and Automation, 1987.
- S. Thrun, “Learning metric-topological maps for indoor mobile robot navigation,” Artificial Intelligence, 1998.
- W. Zhi, L. Ott, R. Senanayake, and F. Ramos, “Continuous occupancy map fusion with fast bayesian hilbert maps,” in International Conference on Robotics and Automation (ICRA), 2019.
- F. Ramos and L. Ott, “Hilbert maps: Scalable continuous occupancy mapping with stochastic gradient descent,” International Journal Robotics Research, 2016.
- W. Zhi, R. Senanayake, L. Ott, and F. Ramos, “Spatiotemporal learning of directional uncertainty in urban environments with kernel recurrent mixture density networks,” IEEE Robotics and Automation Letters, 2019.
- J. J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove, “Deepsdf: Learning continuous signed distance functions for shape representation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
- B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, “Nerf: Representing scenes as neural radiance fields for view synthesis,” in ECCV, 2020.
- T. Zhang and M. Johnson-Roberson, “Beyond nerf underwater: Learning neural reflectance fields for true color correction of marine imagery,” IEEE Robotics and Automation Letters, vol. 8, no. 10, pp. 6467–6474, 2023.
- T. Müller, A. Evans, C. Schied, and A. Keller, “Instant neural graphics primitives with a multiresolution hash encoding,” ACM Trans. Graph., 2022.
- T. Lai, W. Zhi, T. Hermans, and F. Ramos, “Parallelised diffeomorphic sampling-based motion planning,” in Conference on Robot Learning (CoRL), 2021.
- N. Ratliff, M. Zucker, J. A. Bagnell, and S. Srinivasa, “Chomp: Gradient optimization techniques for efficient motion planning,” in IEEE International Conference on Robotics and Automation, 2009.
- W. Zhi, I. Akinola, K. van Wyk, N. Ratliff, and F. Ramos, “Global and reactive motion generation with geometric fabric command sequences,” in IEEE International Conference on Robotics and Automation, ICRA, 2023.
- Q. Yu, G. Wang, X. Hua, S. Zhang, L. Song, J. Zhang, and K. Chen, “Base position optimization for mobile painting robot manipulators with multiple constraints,” Robotics and Computer-Integrated Manufacturing, vol. 54, pp. 56–64, 2018.
- J. J. Yang, W. Yu, J. Kim, and K. Abdel-Malek, “On the placement of open-loop robotic manipulators for reachability,” Mechanism and Machine Theory, vol. 44, no. 4, pp. 671–684, 2009.
- Q. Fan, Z. Gong, B. Tao, Y. Gao, Z. Yin, and H. Ding, “Base position optimization of mobile manipulators for machining large complex components,” Robotics and Computer-Integrated Manufacturing, vol. 70, p. 102138, 2021.
- M. Gadaleta, G. Berselli, and M. Pellicciari, “Energy-optimal layout design of robotic work cells: Potential assessment on an industrial case study,” Robotics and Computer-Integrated Manufacturing, vol. 47, pp. 102–111, 2017, sI: FAIM 2015.
- N. Vahrenkamp, T. Asfour, and R. Dillmann, “Robot placement based on reachability inversion,” in 2013 IEEE International Conference on Robotics and Automation, 2013, pp. 1970–1975.
- A. Makhal and A. K. Goins, “Reuleaux: Robot base placement by reachability analysis,” in 2018 Second IEEE International Conference on Robotic Computing (IRC), 2018, pp. 137–142.
- D. Berenson, J. Kuffner, and H. Choset, “An optimization approach to planning for mobile manipulation,” in 2008 IEEE International Conference on Robotics and Automation, 2008, pp. 1187–1192.
- M. U. Gutmann and A. Hyvärinen, “Noise-contrastive estimation: A new estimation principle for unnormalized statistical models,” in International Conference on Artificial Intelligence and Statistics, 2010.
- I. Loshchilov and F. Hutter, “Decoupled weight decay regularization,” 2019.
- S. Bubeck, “Convex optimization: Algorithms and complexity,” Found. Trends Mach. Learn., 2015.
- E. Coumans and Y. Bai, “Pybullet, a python module for physics simulation for games, robotics and machine learning,” http://pybullet.org, 2016–2019.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.