Papers
Topics
Authors
Recent
Search
2000 character limit reached

Automated Assessment and Adaptive Multimodal Formative Feedback Improves Psychomotor Skills Training Outcomes in Quadrotor Teleoperation

Published 24 May 2024 in cs.RO and cs.HC | (2405.15982v1)

Abstract: The workforce will need to continually upskill in order to meet the evolving demands of industry, especially working with robotic and autonomous systems. Current training methods are not scalable and do not adapt to the skills that learners already possess. In this work, we develop a system that automatically assesses learner skill in a quadrotor teleoperation task using temporal logic task specifications. This assessment is used to generate multimodal feedback based on the principles of effective formative feedback. Participants perceived the feedback positively. Those receiving formative feedback viewed the feedback as more actionable compared to receiving summary statistics. Participants in the multimodal feedback condition were more likely to achieve a safe landing and increased their safe landings more over the experiment compared to other feedback conditions. Finally, we identify themes to improve adaptive feedback and discuss and how training for complex psychomotor tasks can be integrated with learning theories.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (49)
  1. An Analysis of Education and Training Programs in Advanced Manufacturing Using Robotics. RAND Corporation, Pittsburgh, Pennsylvania. https://doi.org/10.7249/RR4244
  2. A systematic review of the role of learning analytics in enhancing feedback practices in higher education. Educational Research Review 37 (Nov. 2022), 100489. https://doi.org/10.1016/j.edurev.2022.100489
  3. D. A. Bell. 1985. Employment skills for the robot age. Robotica 3, 2 (April 1985), 93–95. https://doi.org/10.1017/S0263574700001788
  4. MASCARET: pedagogical multi-agents systems for virtual environment for training. In Proceedings. 2003 International Conference on Cyberworlds. IEEE Comput. Soc, Singapore, 423–430. https://doi.org/10.1109/CYBER.2003.1253485
  5. Deborah L. Butler and Philip H. Winne. 1995. Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research 65, 3 (1995), 245–281. https://doi.org/10.2307/1170684 Place: US Publisher: American Educational Research Assn.
  6. The Importance of Automated Real-Time Performance Feedback in Virtual Reality Temporal Bone Surgery Training. In Proceedings of the 2019 Artificial Intelligence in Education Conference, Seiji Isotani, Eva Millán, Amy Ogan, Peter Hastings, Bruce McLaren, and Rose Luckin (Eds.). Springer International Publishing, Cham, 96–109.
  7. Keep Me in the Loop: Real-Time Feedback with Multimodal Data. Int J Artif Intell Educ 32, 4 (Dec. 2022), 1093–1118. https://doi.org/10.1007/s40593-021-00281-z
  8. Alexandre Donzé and Oded Maler. 2010. Robust Satisfaction of Temporal Logic over Real-Valued Signals. In FORMATS. Lecture Notes in Computer Science, Vol. 6246. Springer, 92–106.
  9. Jon Echeverria and Olga C. Santos. 2021. KUMITRON: Artificial Intelligence System to Monitor Karate Fights that Synchronize Aerial Images with Physiological and Inertial Signals. In Companion Proceedings of the 26th International Conference on Intelligent User Interfaces (, College Station, TX, USA,) (IUI ’21 Companion). Association for Computing Machinery, New York, NY, USA, 37–39. https://doi.org/10.1145/3397482.3450730
  10. Jon Echeverria and Olga C. Santos. 2021. Toward Modeling Psychomotor Performance in Karate Combats Using Computer Vision Pose Estimation. Sensors 21, 24 (2021), 27 pages. https://doi.org/10.3390/s21248378
  11. Rita A. Gregory and Terrance Ward. 2000. Work Force Characteristics in a Robot Driven Construction Industry. In Proceedings of the 17th IAARC/CIB/IEEE/IFAC/IFR International Symposium on Automation and Robotics in Construction. International Association for Automation and Robotics in Construction (IAARC), Taipei, Taiwan, 1–5. https://doi.org/10.22260/ISARC2000/0072 ISSN: 2413-5844.
  12. Towards skill recognition using eye-hand coordination in industrial production. In Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments (Rhodes, Greece) (PETRA ’19). Association for Computing Machinery, New York, NY, USA, 11–20. https://doi.org/10.1145/3316782.3316784
  13. Thanos Hatziapostolou and Iraklis Paraskakis. 2010. Enhancing the Impact of Formative Feedback on Student Learning Through an Online Feedback System. Electronic Journal of e-Learning 8, 2 (2010), 111 – 122.
  14. Conditions that enable effective feedback. Higher Education Research & Development 38, 7 (Nov. 2019), 1401–1416. https://doi.org/10.1080/07294360.2019.1657807
  15. James Hutson and Jason Ceballos. 2023. Rethinking Education in the Age of AI: The Importance of Developing Durable Skills in the Industry 4.0. Journal of Information Economics 1, 2 (July 2023), 26–35. https://doi.org/10.58567/jie01020002
  16. More Than a Number: A Multi-dimensional Framework For Automatically Assessing Human Teleoperation Skill. In Companion of the 2023 ACM/IEEE International Conference on Human-Robot Interaction. ACM, Stockholm Sweden, 653–657. https://doi.org/10.1145/3568294.3580167
  17. Large Language Models Enable Automated Formative Feedback in Human-Robot Interaction Tasks. In Human – Large Language Model Interaction Workshop. ACM/IEEE, Boulder, CO, 2 pages.
  18. Training Hard Skills in Virtual Reality: Developing a Theoretical Framework for AI-Based Immersive Learning. In AI in Learning: Designing the Future, Hannele Niemi, Roy D. Pea, and Yu Lu (Eds.). Springer International Publishing, Cham, 195–213. https://doi.org/10.1007/978-3-031-09687-7_12
  19. Systematic Review of Intelligent Tutoring Systems for Hard Skills Training in Virtual Reality Environments. International Journal of Technology in Education and Science 6, 2 (May 2022), 178–203. https://doi.org/10.46328/ijtes.348 Number: 2.
  20. Ramona Diana Leon. 2023. Employees’ reskilling and upskilling for industry 5.0: Selecting the best professional development programmes. Technology in Society 75 (Nov. 2023), 102393. https://doi.org/10.1016/j.techsoc.2023.102393
  21. Ling Li. 2022. Reskilling and Upskilling the Future-ready Workforce for Industry 4.0 and Beyond. Inf Syst Front 2022 (July 2022), 16 pages. https://doi.org/10.1007/s10796-022-10308-y
  22. Vanda Luengo and Dima Mufti-Alchawafa. 2013. Target the controls during the problem solving activity, a process to produce adapted epistemic feedbacks in ill- defined domains.. In Formative Feedback in Interactive Learning Environments (FFILE) Conference. Springer, Memphis, USA, 8.
  23. Content-Focused Formative Feedback Combining Achievement, Qualitative and Learning Analytics Data. Education Sciences 13, 10 (Oct. 2023), 1014. https://doi.org/10.3390/educsci13101014
  24. Table Tennis Tutor: Forehand Strokes Classification Based on Multimodal Data and Neural Networks. Sensors 21, 9 (2021), 18 pages. https://doi.org/10.3390/s21093121
  25. The Impact of Artificial Intelligence on Workers’ Skills: Upskilling and Reskilling in Organisations. Informing Science: The International Journal of an Emerging Transdiscipline 26 (Feb. 2023), 039–068. https://www.informingscience.org/Publications/5078
  26. Optimizing piano practice with a utility-based scaffold. arXiv:2106.12937 [cs.HC]
  27. Augmented Reality for Supporting Workers in Human–Robot Collaboration. MTI 7, 4 (April 2023), 40. https://doi.org/10.3390/mti7040040
  28. Identification of barriers and their mitigation strategies for industry 5.0 implementation in emerging economies. International Journal of Production Economics 257 (2023), 108770. https://doi.org/10.1016/j.ijpe.2023.108770
  29. Psychomotor Skills in Medical Ultrasound Imaging. Journal of Ultrasound in Medicine 33, 8 (2014), 1349–1352. https://doi.org/10.7863/ultra.33.8.1349 arXiv:https://onlinelibrary.wiley.com/doi/pdf/10.7863/ultra.33.8.1349
  30. OpenAI. 2023. GPT-4V(ision) technical work and authors. https://openai.com/contributions/gpt-4v/
  31. Alberto Casas Ortiz. 2020. Capturing, Modelling, Analyzing and providing Feedback in Martial Arts with Artificial Intelligence to support Psychomotor Learning Activities. Master’s thesis. Universidad Nacional de Educación a Distancia.
  32. Olga Pishchukhina and Angela Allen. 2021. Supporting learning in large classes: online formative assessment and automated feedback. In 2021 30th Annual Conference of the European Association for Education in Electrical and Information Engineering (EAEEIE). IEEE, Prague, Czech Republic, 1–4. https://doi.org/10.1109/EAEEIE50507.2021.9530953
  33. Exploring raw data transformations on inertial sensor data to model user expertise when learning psychomotor skills. User Model User-Adap Inter 2024 (April 2024), 43 pages. https://doi.org/10.1007/s11257-024-09393-2
  34. AI-Supported XR Training: Personalizing Medical First Responder Training. In AI Technologies and Virtual Reality, Kazumi Nakamatsu, Srikanta Patnaik, and Roumen Kountchev (Eds.). Springer Nature Singapore, Singapore, 343–356.
  35. Experimental Assessment of Human-Robot Teaming for Multi-Step Remote Manipulation with Expert Operators. J. Hum.-Robot Interact. 2023 (Oct. 2023), 3618258. https://doi.org/10.1145/3618258
  36. Intelligent Tutoring Systems based on Virtual Reality for the Electrical Domain. RCS 122, 1 (Dec. 2016), 163–174. https://doi.org/10.13053/rcs-122-1-13
  37. When a robot teaches humans: Automated feedback selection accelerates motor learning. Sci. Robot. 4, 27 (Feb. 2019), eaav1560. https://doi.org/10.1126/scirobotics.aav1560
  38. Objective Assessment of Surgical Skills. In Surgical Robotics: Systems Applications and Visions, Jacob Rosen, Blake Hannaford, and Richard M. Satava (Eds.). Springer US, Boston, MA, 619–649. https://doi.org/10.1007/978-1-4419-1126-1_25
  39. Olga C. Santos. 2016. Training the Body: The Potential of AIED to Support Personalized Motor Skills Learning. Int J Artif Intell Educ 26, 2 (June 2016), 730–755. https://doi.org/10.1007/s40593-016-0103-2
  40. Klaus Schwab and Saadia Zahidi. 2020. The Future of Jobs Report 2020. Technical Report. World Economic Forum. https://www3.weforum.org/docs/WEF_Future_of_Jobs_2020.pdf
  41. Formative Assessment of Performance in Diagnostic Ultrasound Using Simulation and Quantitative and Objective Metrics. Military Medicine 184, Supplement_1 (March 2019), 386–391. https://doi.org/10.1093/milmed/usy388
  42. Valerie J. Shute. 2008. Focus on Formative Feedback. Review of Educational Research 78, 1 (2008), 153–189. https://doi.org/10.3102/0034654307313795
  43. Common metrics for human-robot interaction. In Proceedings of the 1st ACM SIGCHI/SIGART conference on Human-robot interaction. ACM, Salt Lake City Utah USA, 33–40. https://doi.org/10.1145/1121241.1121249
  44. Toward Intuitive Teleoperation in Surgery: Human-Centric Evaluation of Teleoperation Algorithms for Robotic Needle Steering. In 2018 IEEE International Conference on Robotics and Automation (ICRA). IEEE, Brisbane, QLD, 5799–5806. https://doi.org/10.1109/ICRA.2018.8460729
  45. Automated Scoring of Complex Tasks in Computer-Based Testing. Lawrence Erlbaum, Mahwah, New Jersey.
  46. An Adaptive Human-Robotic Interaction Architecture for Augmenting Surgery Performance Using Real-Time Workload Sensing—Demonstration of a Semi-autonomous Suction Tool. Hum Factors 66, 4 (April 2024), 1081–1102. https://doi.org/10.1177/00187208221129940 Publisher: SAGE Publications Inc.
  47. Using Reward Shaping to Train Cognitive–based Control Policies for Intelligent Tutoring Systems. In 2024 American Control Conference (ACC). IEEE, Toronto, ON, 8 pages.
  48. Classification of Human Learning Stages via Kernel Distribution Embeddings. IEEE Open Journal of Control Systems 3 (2024), 102–117. https://doi.org/10.1109/OJCSYS.2023.3348704
  49. Vladimir Zotov and Eric Kramkowski. 2023. Moving-Target Intelligent Tutoring System for Marksmanship Training. Int J Artif Intell Educ 33, 4 (Dec. 2023), 817–842. https://doi.org/10.1007/s40593-022-00308-z

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

Sign Up for Free