Papers
Topics
Authors
Recent
Search
2000 character limit reached

Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approach

Published 2 Jul 2024 in cs.RO and cs.LG | (2407.02231v1)

Abstract: This study presents a novel methodology incorporating safety constraints into a robotic simulation during the training of deep reinforcement learning (DRL). The framework integrates specific parts of the safety requirements, such as velocity constraints, as specified by ISO 10218, directly within the DRL model that becomes a part of the robot's learning algorithm. The study then evaluated the efficiency of these safety constraints by subjecting the DRL model to various scenarios, including grasping tasks with and without obstacle avoidance. The validation process involved comprehensive simulation-based testing of the DRL model's responses to potential hazards and its compliance. Also, the performance of the system is carried out by the functional safety standards IEC 61508 to determine the safety integrity level. The study indicated a significant improvement in the safety performance of the robotic system. The proposed DRL model anticipates and mitigates hazards while maintaining operational efficiency. This study was validated in a testbed with a collaborative robotic arm with safety sensors and assessed with metrics such as the average number of safety violations, obstacle avoidance, and the number of successful grasps. The proposed approach outperforms the conventional method by a 16.5% average success rate on the tested scenarios in the simulations and 2.5% in the testbed without safety violations. The project repository is available at https://github.com/ammar-n-abbas/sim2real-ur-gym-gazebo.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (42)
  1. C. Li, P. Zheng, P. Zhou, Y. Yin, C. K. Lee, and L. Wang, “Unleashing mixed-reality capability in deep reinforcement learning-based robot motion generation towards safe human–robot collaboration,” Journal of Manufacturing Systems, vol. 74, pp. 411–421, 2024.
  2. J. Garcıa and F. Fernández, “A comprehensive survey on safe reinforcement learning,” Journal of Machine Learning Research, vol. 16, no. 1, pp. 1437–1480, 2015.
  3. J. Thumm and M. Althoff, “Provably safe deep reinforcement learning for robotic manipulation in human environments,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 6344–6350.
  4. W. Qian, Z. Xia, J. Xiong, Y. Gan, Y. Guo, S. Weng, H. Deng, Y. Hu, and J. Zhang, “Manipulation task simulation using ros and gazebo,” in 2014 IEEE International Conference on Robotics and Biomimetics (ROBIO 2014).   IEEE, 2014, pp. 2594–2598.
  5. International Organization for Standardization, “Robots and robotic devices-safety requirements for industrial robots-part 1: Robots,” Standard DIN EN ISO 10218–1, Jan 2012, ISO 10218–1:2011.
  6. DIN German Institute for Standardization, “Functional safety of electrical/electronic/programmable electronic safety-related systems part 1: General requirements,” Standard DIN EN 61508–1, Feb 2016, IEC 61508–1:2010.
  7. X. Tong and W. Lei, “A systematic analysis of functional safety certification practices in industrial robot software development,” in MATEC Web of Conferences, vol. 100.   EDP Sciences, 2017, p. 02011.
  8. B. Chen, “Verification and validation strategies on collaborative robotic agv: the process and challenges in the certification development,” in 2017 IEEE International Symposium on Product Safety and Compliance Engineering-Taiwan (ISPCE-TW).   IEEE, 2017, pp. 1–2.
  9. H. Krasowski, J. Thumm, M. Müller, L. Schäfer, X. Wang, and M. Althoff, “Provably safe reinforcement learning: Conceptual analysis, survey, and benchmarking,” Transactions on Machine Learning Research, 2023.
  10. S. Lakshminarayanan, S. Kana, A. De San Bernabe, S. H. Turlapati, D. Accoto, and D. Campolo, “Robots in manufacturing: Programming, control, and safety standards,” in Digital Manufacturing.   Elsevier, 2024, pp. 85–131.
  11. E. Marchesini, D. Corsi, and A. Farinelli, “Exploring safer behaviors for deep reinforcement learning,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 36, no. 7, 2022, pp. 7701–7709.
  12. H.-L. Hsu, Q. Huang, and S. Ha, “Improving safety in deep reinforcement learning using unsupervised action planning,” in 2022 International Conference on Robotics and Automation (ICRA).   IEEE, 2022, pp. 5567–5573.
  13. L. Brunke, M. Greeff, A. W. Hall, Z. Yuan, S. Zhou, J. Panerati, and A. P. Schoellig, “Safe learning in robotics: From learning-based control to safe reinforcement learning,” Annual Review of Control, Robotics, and Autonomous Systems, vol. 5, pp. 411–444, 2022.
  14. Y. S. Shao, C. Chen, S. Kousik, and R. Vasudevan, “Reachability-based trajectory safeguard (rts): A safe and fast reinforcement learning safety layer for continuous control,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 3663–3670, 2021.
  15. K. Fan, Z. Chen, G. Ferrigno, and E. De Momi, “Learn from safe experience: Safe reinforcement learning for task automation of surgical robot,” IEEE Transactions on Artificial Intelligence, 2024.
  16. M. El-Shamouty, X. Wu, S. Yang, M. Albus, and M. F. Huber, “Towards safe human-robot collaboration using deep reinforcement learning,” in 2020 IEEE international conference on robotics and automation (ICRA).   IEEE, 2020, pp. 4899–4905.
  17. B. Sangiovanni, G. P. Incremona, M. Piastra, and A. Ferrara, “Self-configuring robot path planning with obstacle avoidance via deep reinforcement learning,” IEEE Control Systems Letters, vol. 5, no. 2, pp. 397–402, 2020.
  18. J. Kim, J. hyeon Park, D. Cho, and H. J. Kim, “Automating reinforcement learning with example-based resets,” IEEE Robotics and Automation Letters, vol. 7, no. 3, pp. 6606–6613, 2022.
  19. S. Lange, M. Riedmiller, and A. Voigtländer, “Autonomous reinforcement learning on raw visual input data in a real world application,” in The 2012 international joint conference on neural networks (IJCNN).   IEEE, 2012, pp. 1–8.
  20. A. Gupta, J. Yu, T. Z. Zhao, V. Kumar, A. Rovinsky, K. Xu, T. Devlin, and S. Levine, “Reset-free reinforcement learning via multi-task learning: Learning dexterous manipulation behaviors without human intervention,” in 2021 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2021, pp. 6664–6671.
  21. G. Thomas, Y. Luo, and T. Ma, “Safe reinforcement learning by imagining the near future,” Advances in Neural Information Processing Systems, vol. 34, pp. 13 859–13 869, 2021.
  22. N. Hunt, N. Fulton, S. Magliacane, T. N. Hoang, S. Das, and A. Solar-Lezama, “Verifiably safe exploration for end-to-end reinforcement learning,” in Proceedings of the 24th International Conference on Hybrid Systems: Computation and Control, 2021, pp. 1–11.
  23. P. Liu, K. Zhang, D. Tateo, S. Jauhri, Z. Hu, J. Peters, and G. Chalvatzaki, “Safe reinforcement learning of dynamic high-dimensional robotic tasks: navigation, manipulation, interaction,” in 2023 IEEE International Conference on Robotics and Automation (ICRA).   IEEE, 2023, pp. 9449–9456.
  24. S. Zhang, D.-H. Zhai, Y. Xiong, J. Lin, and Y. Xia, “Safety-critical control for robotic systems with uncertain model via control barrier function,” International Journal of Robust and Nonlinear Control, vol. 33, no. 6, pp. 3661–3676, 2023.
  25. M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu, “Safe reinforcement learning via shielding,” in Proceedings of the AAAI conference on artificial intelligence, vol. 32, no. 1, 2018.
  26. N. Jansen, B. Könighofer, S. Junges, A. Serban, and R. Bloem, “Safe reinforcement learning using probabilistic shields,” in 31st International Conference on Concurrency Theory (CONCUR 2020).   Schloss-Dagstuhl-Leibniz Zentrum für Informatik, 2020.
  27. A. Ray, J. Achiam, and D. Amodei, “Benchmarking safe exploration in deep reinforcement learning,” arXiv preprint arXiv:1910.01708, 2019.
  28. Q. Yang, T. D. Simão, S. H. Tindemans, and M. T. Spaan, “Safety-constrained reinforcement learning with a distributional safety critic,” Machine Learning, vol. 112, no. 3, pp. 859–887, 2023.
  29. M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder et al., “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” arXiv preprint arXiv:1802.09464, 2018.
  30. J. Ji, B. Zhang, J. Zhou, X. Pan, W. Huang, R. Sun, Y. Geng, Y. Zhong, J. Dai, and Y. Yang, “Safety gymnasium: A unified safe reinforcement learning benchmark,” Advances in Neural Information Processing Systems, vol. 36, 2023.
  31. C. Saliba, M. K. Bugeja, S. G. Fabri, M. Di Castro, A. Mosca, and M. Ferre, “A training simulator for teleoperated robots deployed at cern.” in ICINCO (2), 2018, pp. 293–300.
  32. J. Yoon, B. Son, and D. Lee, “Comparative study of physics engines for robot simulation with mechanical interaction,” Applied Sciences, vol. 13, no. 2, p. 680, 2023.
  33. M. Connolly, A. K. Ramasubramanian, M. Kelly, J. McEvoy, and N. Papakostas, “Realistic simulation of robotic grasping tasks: review and application,” Procedia CIRP, vol. 104, pp. 1704–1709, 2021.
  34. A. K. Ramasubramanian, M. Connolly, R. Mathew, and N. Papakostas, “Automatic simulation-based design and validation of robotic gripper fingers,” CIRP Annals, vol. 71, no. 1, pp. 137–140, 2022.
  35. M. Plappert, M. Andrychowicz, A. Ray, B. McGrew, B. Baker, G. Powell, J. Schneider, J. Tobin, M. Chociej, P. Welinder, V. Kumar, and W. Zaremba, “Multi-goal reinforcement learning: Challenging robotics environments and request for research,” 2018.
  36. C. C. Beltran-Hernandez, D. Petit, I. G. Ramirez-Alpizar, and K. Harada, “Accelerating robot learning of contact-rich manipulations: A curriculum learning study,” arXiv preprint arXiv:1910.01708, 2022.
  37. K. Katyal, I. Wang, P. Burlina et al., “Leveraging deep reinforcement learning for reaching robotic tasks,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2017, pp. 18–19.
  38. A. N. Abbas, “ammar-n-abbas/sim2real-ur-gym-gazebo: v1.0.0,” Jan. 2024. [Online]. Available: https://doi.org/10.5281/zenodo.10569005
  39. R. Jiang, Z. Wang, B. He, and Z. Di, “Vision-based deep reinforcement learning for ur5 robot motion control,” in 2021 IEEE International Conference on Consumer Electronics and Computer Engineering (ICCECE).   IEEE, 2021, pp. 246–250.
  40. C. Wang, Q. Zhang, Q. Tian, S. Li, X. Wang, D. Lane, Y. Petillot, and S. Wang, “Learning mobile manipulation through deep reinforcement learning,” Sensors, vol. 20, no. 3, p. 939, 2020.
  41. R. de Lazcano, K. Andreas, J. J. Tai, S. R. Lee, and J. Terry, “Gymnasium robotics,” 2023. [Online]. Available: http://github.com/Farama-Foundation/Gymnasium-Robotics
  42. A. Kuznetsov, P. Shvechikov, A. Grishin, and D. Vetrov, “Controlling overestimation bias with truncated mixture of continuous distributional quantile critics,” in International Conference on Machine Learning.   PMLR, 2020, pp. 5556–5566.
Citations (1)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up