Papers
Topics
Authors
Recent
Search
2000 character limit reached

Narrative Review of Emotional Expression Support in XR: Psychophysiology of Speech-to-Text Interfaces

Published 22 May 2024 in cs.HC | (2405.13924v3)

Abstract: This narrative review examines recent advancements, limitations, and research gaps in integrating emotional expression into speech-to-text (STT) interfaces within extended reality (XR) environments. Drawing from 37 peer-reviewed studies published between 2020 and 2024, we synthesized literature across multiple domains, including affective computing, psychophysiology, captioning innovation, and immersive human-computer interaction. Thematic categories include communication enhancement technologies for Deaf and Hard of Hearing (DHH) users, emotive captioning strategies, visual and affective augmentation in AR/VR, speech emotion recognition, and the development of empathic systems. Despite the growing accessibility of real-time STT tools, such systems largely fail to convey affective nuance, limiting the richness of communication for DHH users and other caption consumers. This review highlights emerging approaches such as animated captions, emojilization, color-coded overlays, and avatar-based emotion visualization, but finds a persistent gap in real-time emotion-aware captioning within immersive XR contexts. We identify key research opportunities at the intersection of accessibility, XR, and emotional expression, and propose future directions for the development of affect-responsive, user-centered captioning interfaces.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. A.-W. Harzing, “Publish or perish.” https://harzing.com/resources/publish-or-perish, 2007.
  2. M. Ouzzani, H. Hammady, Z. Fedorowicz, and A. Elmagarmid, “Rayyan — a web and mobile app for systematic reviews,” Systematic Reviews, vol. 5, p. 210, 2016.
  3. F. Loizides, S. Basson, D. Kanevsky, O. Prilepova, S. Savla, and S. Zaraysky, “Breaking boundaries with live transcribe: Expanding use cases beyond standard captioning scenarios,” in Proceedings of the 22nd International ACM SIGACCESS Conference on Computers and Accessibility, pp. 1–6, 2020.
  4. D. Arnold and A. Tremblay, “Interaction of deaf and hearing preschool children,” Journal of Communication Disorders, vol. 12, no. 3, pp. 245–251, 1979.
  5. R. Rashid, V. Quoc, R. Hunt, and D. I. Fels, “Dancing with words: Using animated text for captioning,” Intl. Journal of Human–Computer Interaction, vol. 24, no. 5, pp. 505–519, 2008.
  6. V. Murphy-Berman and L. Whobrey, “The impact of captions on hearing-impaired children’s affective reactions to television,” The Journal of Special Education, vol. 17, no. 1, pp. 47–62, 1983.
  7. R. Rashid, J. Aitken, and D. I. Fels, “Expressing emotions using animated text captions,” in Computers Helping People with Special Needs: 10th International Conference, ICCHP 2006, Linz, Austria, July 11-13, 2006. Proceedings 10, pp. 24–31, Springer, 2006.
  8. D. G. Lee, D. I. Fels, and J. P. Udo, “Emotive captioning,” Computers in Entertainment (CIE), vol. 5, no. 2, p. 11, 2007.
  9. J. Hu, Q. Xu, L. P. Fu, and Y. Xu, “Emojilization: An automated method for speech to emoji-labeled text,” in Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, pp. 1–6, 2019.
  10. R. Poguntke, T. Mantz, M. Hassib, A. Schmidt, and S. Schneegass, “Smile to me: investigating emotions and their representation in text-based messaging in the wild,” in Proceedings of Mensch und Computer 2019, pp. 373–385, ACM, 2019.
  11. Q. Li, “Text vs. images: Understanding emotional expressions on social media during covid-19 pandemic,” Human Factors in Communication of Design, vol. 49, p. 18, 2022.
  12. K. Kilteni, R. Groten, and M. Slater, “The sense of embodiment in virtual reality,” Presence: Virtual and Augmented Reality, vol. 21, no. 4, pp. 373–387, 2012.
  13. S. Lie, K. Røykenes, A. Sæheim, and K. Groven, “Developing a virtual reality educational tool to stimulate emotions for learning: focus group study,” JMIR Formative Research, vol. 7, p. e41829, 2023.
  14. F. Pallavicini and A. Pepe, “Virtual reality games and the role of body involvement in enhancing positive emotions and decreasing anxiety: within-subjects pilot study,” JMIR Serious Games, vol. 8, no. 2, p. e15635, 2020.
  15. A. Alnafjan, A. Aljumaah, H. Alaskar, and R. Alshraihi, “Designing “najeeb”: Technology-enhanced learning for children with impaired hearing using arabic sign-language arsl applications,” in 2017 International Conference on Computer and Applications (ICCA), pp. 238–273, IEEE, 2017.
  16. J. Li, “Real-time augmented reality visual-captions for deaf and hard-of-hearing children in classrooms,” in 2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 641–642, IEEE, 2023.
  17. J. Pirker, J. Kopf, A. Kainz, A. Dengel, and B. Buchbauer, “The potential of virtual reality for computer science education-engaging students through immersive visualizations,” in 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 297–302, IEEE, 2021.
  18. X. Chen and H. Chen, “Emotion recognition using facial expressions in an immersive virtual reality application,” Virtual Reality, vol. 27, no. 3, pp. 1717–1732, 2023.
  19. A. Elor, A. Song, and S. Kurniawan, “Understanding emotional expression with haptic feedback vest patterns and immersive virtual reality,” in 2021 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 183–188, IEEE, 2021.
  20. L. Bartram, A. Patra, and M. Stone, “Affective color in visualization,” in Proceedings of the 2017 CHI conference on human factors in computing systems, pp. 1364–1374, 2017.
  21. D. Cernea, C. Weber, A. Ebert, and A. Kerren, “Emotion-prints: Interaction-driven emotion visualization on multi-touch interfaces,” in Visualization and Data Analysis 2015, vol. 9397, pp. 82–96, SPIE, 2015.
  22. H. Wang, H. Prendinger, and T. Igarashi, “Communicating emotions in online chat using physiological sensors and animated text,” in CHI’04 extended abstracts on Human factors in computing systems, pp. 1171–1174, 2004.
  23. D. J. Schiano, S. M. Ehrlich, K. Rahardja, and K. Sheridan, “Face to interface: facial affect in (hu) man and machine,” in Proceedings of the SIGCHI conference on Human factors in computing systems, pp. 193–200, 2000.
  24. D. Kumarapeli, S. Jung, and R. W. Lindeman, “Emotional avatars: Effect of uncanniness in identifying emotions using avatar expressions,” in 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW), pp. 650–651, IEEE, 2022.
  25. M. Rebol, C. Güti, and K. Pietroszek, “Passing a non-verbal turing test: Evaluating gesture animations generated from speech,” in 2021 IEEE Virtual Reality and 3D User Interfaces (VR), pp. 573–581, IEEE, 2021.
  26. E. Bosch, D. Bethge, M. Klosterkamp, and T. Kosch, “Empathic technologies shaping innovative interaction: Future directions of affective computing,” in Adjunct Proceedings of the 2022 Nordic Human-Computer Interaction Conference, pp. 1–3, 2022.
  27. K. Daher, J. Casas, O. A. Khaled, and E. Mugellini, “Empathic chatbot response for medical assistance,” in Proceedings of the 20th ACM International Conference on Intelligent Virtual Agents, pp. 1–3, 2020.
  28. X. Sun and H. Guan, “Research on empathic remediation mechanism of chatbots mediated by social presence and trust,” in Proceedings of the 2022 6th International Conference on Electronic Information Technology and Computer Engineering, pp. 772–776, 2022.
  29. J. Casas, T. Spring, K. Daher, E. Mugellini, O. A. Khaled, and P. Cudré-Mauroux, “Enhancing conversational agents with empathic abilities,” in Proceedings of the 21st ACM International Conference on Intelligent Virtual Agents, pp. 41–47, 2021.
  30. J.-Y. Jung and A. Bozzon, “Are female chatbots more empathic?-discussing gendered conversational agent through empathic design,” in Proceedings of the 2nd Empathy-Centric Design Workshop, pp. 1–5, 2023.
  31. A. Roshdy, S. Al Kork, A. Karar, A. Al Sabi, Z. Al Barakeh, F. ElSayed, T. Beyrouthy, and A. Nait-Ali, “Machine empathy: Digitizing human emotions,” in 2021 International Symposium on Electrical, Electronics and Information Engineering, pp. 307–311, 2021.
  32. N. Saffaryazdi, Y. Goonesekera, N. Saffaryazdi, N. D. Hailemariam, E. G. Temesgen, S. Nanayakkara, E. Broadbent, and M. Billinghurst, “Emotion recognition in conversations using brain and physiological signals,” in 27th International Conference on Intelligent User Interfaces, pp. 229–242, 2022.
  33. E. Bagheri, P. G. Esteban, H.-L. Cao, A. D. Beir, D. Lefeber, and B. Vanderborght, “An autonomous cognitive empathy model responsive to users’ facial emotion expressions,” ACM Transactions on Interactive Intelligent Systems (TIIS), vol. 10, no. 3, pp. 1–23, 2020.
  34. T. Lin, L. Huang, B. Hannaford, C. Tran, J. Raiti, R. Zaragoza, T. Feng, L. Wagner, and J. James, “Empathics system: application of emotion analysis ai through smart glasses,” in Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments, pp. 1–4, 2020.
  35. K. Gupta, S. W. Chan, Y. S. Pai, A. Sumich, S. Nanayakkara, and M. Billinghurst, “Towards understanding physiological responses to emotional autobiographical memory recall in mobile vr scenarios,” in Adjunct Publication of the 23rd International Conference on Mobile Human-Computer Interaction, pp. 1–5, 2021.
  36. D. Higgins, Y. Zhan, B. R. Cowan, and R. McDonnell, “Investigating the effect of visual realism on empathic responses to emotionally expressive virtual humans,” in ACM Symposium on Applied Perception 2023, pp. 1–7, 2023.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 0 likes about this paper.

Sign Up for Free