Papers
Topics
Authors
Recent
Search
2000 character limit reached

Designing for Harm Reduction: Communication Repair for Multicultural Users' Voice Interactions

Published 1 Mar 2024 in cs.HC and cs.CY | (2403.00265v1)

Abstract: Voice assistants' inability to serve people-of-color and non-native English speakers has largely been documented as a quality-of-service harm. However, little work has investigated what downstream harms propagate from this poor service. How does poor usability materially manifest and affect users' lives? And what interaction designs might help users recover from these effects? We identify 6 downstream harms that propagate from quality-of-service harms in voice assistants. Through interviews and design activities with 16 multicultural participants, we unveil these 6 harms, outline how multicultural users uniquely personify their voice assistant, and suggest how these harms and personifications may affect their interactions. Lastly, we employ techniques from psychology on communication repair to contribute suggestions for harm-reducing repair that may be implemented in voice technologies. Our communication repair strategies include: identity affirmations (intermittent frequency), cultural sensitivity, and blame redirection. This work shows potential for a harm-repair framework to positively influence voice interactions.

Authors (2)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (95)
  1. From dissonance to disidentification: Selectivity in the self-affirmation process. Journal of Personality and Social Psychology 68, 6 (1995), 986.
  2. Shaowen Bardzell and Jeffrey Bardzell. 2011. Towards a feminist HCI methodology: social science, feminism, and HCI. In Proceedings of the SIGCHI conference on human factors in computing systems. 675–684.
  3. Stereotype threat and working memory: mechanisms, alleviation, and spillover. Journal of Experimental Psychology: General 136, 2 (2007), 256.
  4. Communication breakdowns between families and Alexa. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–13.
  5. Ruha Benjamin. 2020. Race after technology: Abolitionist tools for the new Jim code.
  6. Patient and consumer safety risks when using conversational assistants for medical information: an observational study of Siri, Alexa, and Google Assistant. Journal of medical Internet research 20, 9 (2018), e11510.
  7. Once a kind friend is now a thing: Understanding how conversational agents at home are forgotten. In Proceedings of the 2019 on Designing Interactive Systems Conference. 1557–1569.
  8. Toward a Multilingual Conversational Agent: Challenges and Expectations of Code-mixing Multilingual Users. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17.
  9. Geoffrey L Cohen and David K Sherman. 2014. The psychology of change: Self-affirmation and social psychological intervention. Annual review of psychology 65 (2014), 333–371.
  10. Voice-activated virtual home assistant use and social isolation and loneliness among older adults: mini review. Frontiers in Public Health 9 (2021), 742012.
  11. My bad! repairing intelligent voice assistant errors improves interaction. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–24.
  12. Conversational assistants and gender stereotypes: Public perceptions and desiderata for voice personas. In Proceedings of the second workshop on gender bias in natural language processing. 72–78.
  13. Tales Rebequi Costa Borges de Souza and João Luiz Bernardes. 2016. The influences of culture on user experience. In Cross-Cultural Design: 8th International Conference, CCD 2016, Held as Part of HCI International 2016, Toronto, ON, Canada, July 17-22, 2016, Proceedings 8. Springer, 43–52.
  14. Smit Desai and Michael Twidale. 2022. Is Alexa like a computer? A search engine? A friend? A silly child? Yes.. In Proceedings of the 4th Conference on Conversational User Interfaces. 1–4.
  15. Mapping perceptions of humanness in intelligent personal assistant interaction. In Proceedings of the 21st international conference on human-computer interaction with mobile devices and services. 1–12.
  16. Rashmi Dyal-Chand. 2021. Autocorrecting for Whiteness. BUL Rev. 101 (2021), 191.
  17. On seeing human: a three-factor theory of anthropomorphism. Psychological review 114, 4 (2007), 864.
  18. Alexa, my love: Analyzing reviews of amazon echo. In 2018 IEEE SmartWorld, Ubiquitous Intelligence & Computing, Advanced & Trusted Computing, Scalable Computing & Communications, Cloud & Big Data Computing, Internet of People and Smart City Innovation (SmartWorld/SCALCOM/UIC/ATC/CBDCom/IOP/SCI). IEEE, 372–380.
  19. Elizabeth Gerber. 2011. Tech break up: a research method for understanding people’s attachment to their technology. In Proceedings of the 8th ACM Conference on Creativity and Cognition. 137–146.
  20. Children’s mental, social, and moral attributions toward a familiar digital voice assistant. Human Behavior and Emerging Technologies 3, 5 (2021), 1118–1131.
  21. Ben Green and Salomé Viljoen. 2020. Algorithmic realism: expanding the boundaries of algorithmic thought. In Proceedings of the 2020 conference on fairness, accountability, and transparency. 19–31.
  22. Andrea L Guzman. 2019. Voices in and of the machine: Source orientation toward mobile virtual assistants. Computers in Human Behavior 90 (2019), 343–350.
  23. Keep it Short: A Comparison of Voice Assistants’ Response Behavior. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–12.
  24. Lois M Haggard and Daniel R Williams. 1992. Identity affirmation through leisure activities: Leisure symbols of the self. Journal of leisure research 24, 1 (1992), 1–18.
  25. Bruce Hanington and Bella Martin. 2012. Universal methods of design: 100 ways to research complex problems. Develop innovative ideas, and design effective solutions: Rockport Publishers (2012).
  26. “It’s Kind of Like Code-Switching”: Black Older Adults’ Experiences with a Voice Assistant for Health Information Seeking. In CHI Conference on Human Factors in Computing Systems. 1–15.
  27. Fan He and Catherine M Burns. 2022. A Battle of Voices: A Study of the Relationship between Driving Experience, Driving Style, and In-Vehicle Voice Assistant Character. In Proceedings of the 14th International Conference on Automotive User Interfaces and Interactive Vehicular Applications. 236–242.
  28. Geert Hofstede. 2011. Dimensionalizing cultures: The Hofstede model in context. Online readings in psychology and culture 2, 1 (2011), 8.
  29. Inequity in Popular Speech Recognition Systems for Accented English Speech. In 27th International Conference on Intelligent User Interfaces. 66–68.
  30. Intelligence Insider. 2023. Voice Assistants in 2023: Usage, growth, and future of the AI voice assistant market. https://www.insiderintelligence.com/insights/voice-assistants/
  31. The language experience and proficiency questionnaire (LEAP-Q): Ten years later. Bilingualism: Language and Cognition 23, 5 (2020), 945–950.
  32. Just say no (to stereotyping): effects of training in the negation of stereotypic associations on stereotype activation. Journal of personality and social psychology 78, 5 (2000), 871.
  33. ” Why did this voice agent not understand me?” error recovery strategy for in-vehicle voice user interface. In Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications: Adjunct Proceedings. 146–150.
  34. Sunyoung Kim and Abhishek Choudhury. 2021. Exploring older adults’ perception and use of smart speaker-based voice assistants: A longitudinal study. Computers in Human Behavior 124 (2021), 106914.
  35. Racial disparities in automated speech recognition. Proceedings of the National Academy of Sciences 117, 14 (2020), 7684–7689.
  36. Genie in the bottle: Anthropomorphized perceptions of conversational agents. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13.
  37. Cross-cultural user experience design helping product designers to consider cultural differences. In Cross-Cultural Design Methods, Practice and Impact: 7th International Conference, CCD 2015, Held as Part of HCI International 2015, Los Angeles, CA, USA, August 2-7, 2015, Proceedings, Part I 7. Springer, 58–70.
  38. Deepfakes, Phrenology, Surveillance, and More! A Taxonomy of AI Privacy Risks. arXiv preprint arXiv:2310.07879 (2023).
  39. Empirical analysis of bias in voice-based personal assistants. In Companion Proceedings of the 2019 World Wide Web Conference. 533–538.
  40. Fanjue Liu. 2023. Hanging out with my pandemic pal: Contextualizing motivations of anthropomorphizing voice assistants during covid-19. Journal of Promotion Management 29, 5 (2023), 676–704.
  41. Chris Loersch and Nathan L Arbuckle. 2013. Unraveling the mystery of music: music as an evolved group process. Journal of Personality and Social Psychology 105, 5 (2013), 777.
  42. Irene Lopatovska and Harriet Williams. 2018. Personification of the Amazon Alexa: BFF or a mindless companion. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. 265–268.
  43. Owning mistakes sincerely: Strategies for mitigating AI errors. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems. 1–11.
  44. Aaron Marcus. 2006. Cross-cultural user-experience design. In International Conference on Theory and Application of Diagrams. Springer, 16–24.
  45. Aaron Marcus. 2014. Cross-cultural user-experience design for work, home, play, and on the way. In SIGGRAPH Asia 2014 Courses. 1–124.
  46. The Language Experience and Proficiency Questionnaire (LEAP-Q): Assessing language profiles in bilinguals and multilinguals. (2007).
  47. Combating stereotype threat: The effect of self-affirmation on women’s intellectual performance. Journal of Experimental Social Psychology 42, 2 (2006), 236–243.
  48. Sean M McCrea and Edward R Hirt. 2011. Limitations on the Substitutability of Self-Protective Processes. Social Psychology (2011).
  49. Graeme McLean and Kofi Osei-Frimpong. 2019. Hey Alexa… examine the variables influencing the use of artificial intelligent in-home voice assistants. Computers in Human Behavior 99 (2019), 28–37.
  50. The ‘Reading the mind in the Eyes’ test and emotional intelligence. Royal Society Open Science 7, 9 (2020), 201305.
  51. “I don’t Think These Devices are Very Culturally Sensitive.”—Impact of Automated Speech Recognition Errors on African Americans. Frontiers in Artificial Intelligence (2021), 169.
  52. How perceptions of intelligence and anthropomorphism affect adoption of personal intelligent agents. Electronic Markets 31 (2021), 343–364.
  53. Joel T Nadler and MH Clark. 2011. Stereotype threat: A Meta-Analysis comparing african americans to hispanic americans 1. Journal of Applied Social Psychology 41, 4 (2011), 872–890.
  54. Computers are social actors. In Proceedings of the SIGCHI conference on Human factors in computing systems. 72–78.
  55. Differences in interactions with a conversational agent. International journal of environmental research and public health 17, 9 (2020), 3189.
  56. Josh Packard. 2008. ‘I’m gonna show you what it’s really like out here’: the power and limitation of participatory visual methods. Visual studies 23, 1 (2008), 63–77.
  57. User experience with smart voice assistants: The accent perspective. In 2019 10th international conference on computing, communication and networking technologies (ICCCNT). IEEE, 1–6.
  58. Do you understand the words that are comin outta my mouth? Voice assistant comprehension of medication names. NPJ digital medicine 2, 1 (2019), 1–6.
  59. Dasom Park and Kiechan Namkung. 2021. Exploring users’ mental models for anthropomorphized voice assistants through psychological approaches. Applied Sciences 11, 23 (2021), 11147.
  60. Elizabeth Petrick. 2019. Curb Cuts and Computers: Advocating for Design Equality in the 1980s. Design Issues 35, 4 (2019), 23–32.
  61. Voice interfaces in everyday life. In proceedings of the 2018 CHI conference on human factors in computing systems. 1–12.
  62. ” Phantom Friend” or” Just a Box with Information” Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–21.
  63. ” Alexa is my new BFF” Social Roles, User Satisfaction, and Personification of the Amazon Echo. In Proceedings of the 2017 CHI conference extended abstracts on human factors in computing systems. 2853–2859.
  64. Aung Pyae and Paul Scifleet. 2018. Investigating differences between native English and non-native English speakers in interacting with a voice user interface: A case of Google Home. In Proceedings of the 30th Australian conference on computer-human interaction. 548–553.
  65. Speaking from Experience: Trans/Non-Binary Requirements for Voice-Activated AI. Proceedings of the ACM on Human-Computer Interaction 5, CSCW1 (2021), 1–27.
  66. Multiple social identities and stereotype threat: imbalance, accessibility, and working memory. Journal of personality and social psychology 96, 5 (2009), 949.
  67. Johnny Saldaña. 2021. The coding manual for qualitative researchers. sage.
  68. Cate C Samuelson and Elizabeth Litzler. 2016. Community cultural wealth: An assets-based approach to persistence of engineering students of color. Journal of Engineering Education 105, 1 (2016), 93–117.
  69. A framework of severity for harmful content online. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–33.
  70. Florian Schneider and Justus Hagmann. 2022. Assisting the Assistant: How and Why People Show Reciprocal Behavior Towards Voice Assistants. In International Conference on Human-Computer Interaction. Springer, 566–579.
  71. Servant, friend or master? The relationships users build with voice-controlled smart devices. Journal of Marketing Management 35, 7-8 (2019), 693–715.
  72. William Seymour and Max Van Kleek. 2021. Exploring interactions between trust, anthropomorphism, and relationship development in voice assistants. Proceedings of the ACM on Human-Computer Interaction 5, CSCW2 (2021), 1–16.
  73. Chun Shao and K Hazel Kwon. 2021. Hello Alexa! Exploring effects of motivational factors and social presence on satisfaction with artificial intelligence-enabled gadgets. Human Behavior and Emerging Technologies 3, 5 (2021), 978–988.
  74. Jenessa R Shapiro and Steven L Neuberg. 2007. From stereotype threat to stereotype threats: Implications of a multi-threat framework for causes, moderators, mediators, consequences, and interventions. Personality and Social Psychology Review 11, 2 (2007), 107–130.
  75. Sociotechnical harms: scoping a taxonomy for harm reduction. arXiv preprint arXiv:2210.05791 (2022).
  76. David K Sherman and Geoffrey L Cohen. 2006. The psychology of self-defense: Self-affirmation theory. Advances in experimental social psychology 38 (2006), 183–242.
  77. Affirmed yet unaware: exploring the role of awareness in the process of self-affirmation. Journal of personality and social psychology 97, 5 (2009), 745.
  78. Claude M Steele and Joshua Aronson. 1995. Stereotype threat and the intellectual test performance of African Americans. Journal of personality and social psychology 69, 5 (1995), 797.
  79. Understanding psychological reactance. Zeitschrift für Psychologie (2015).
  80. ” All in the Same Boat”: Tradeoffs of Voice Assistant Ownership for Mixed-Visual-Ability Families. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
  81. Yolande Strengers and Jenny Kennedy. 2021. The smart wife: Why Siri, Alexa, and other smart home devices need a feminist reboot. Mit Press.
  82. Servant by default? How humans perceive their relationship with conversational AI. Cyberpsychology: Journal of Psychosocial Research on Cyberspace 17, 3 (2023).
  83. ‘It’s not a romantic relationship’: Stories of Adoption and Abandonment of Smart Speakers at Home. In Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia. 71–82.
  84. Is it human? The role of anthropomorphism as a driver for the successful acceptance of digital voice assistants. (2019).
  85. Sandra Weber. 2008. Visual images in research. Handbook of the arts in qualitative research (2008), 41–53.
  86. Taxonomy of risks posed by language models. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency. 214–229.
  87. Can Voice Assistants Be Microaggressors? Cross-Race Psychological Responses to Failures of Automatic Speech Recognition. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–14.
  88. Kimi Wenzel and Geoff Kaufman. 2023. Challenges in Designing Racially Inclusive Language Technologies. arXiv preprint arXiv:2303.13546 (2023).
  89. Harry F Wolcott. 1994. Transforming qualitative data: Description, analysis, and interpretation. Sage.
  90. The consequences of chronic stereotype threat: domain disidentification and abandonment. Journal of personality and social psychology 103, 4 (2012), 635.
  91. Heather Suzanne Woods. 2018. Asking more of Siri and Alexa: feminine persona in service of surveillance capitalism. Critical Studies in Media Communication 35, 4 (2018), 334–349.
  92. The Slow Violence of Surveillance Capitalism: How Online Behavioral Advertising Harms People. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency. 1826–1837.
  93. See what I’m saying? Comparing intelligent personal assistant use for native and non-native language speakers. In 22nd international conference on human-computer interaction with mobile devices and services. 1–9.
  94. Charles Yiu. 2013. UX design with international teams: Challenges and best practices. In CHI’13 Extended Abstracts on Human Factors in Computing Systems. 2333–2336.
  95. Shoshana Zuboff. 2023. The age of surveillance capitalism. In Social Theory Re-Wired. Routledge, 203–213.
Citations (6)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 2 tweets with 55 likes about this paper.

Sign Up for Free