A Diachronic Perspective on User Trust in AI under Uncertainty

Abstract: In a human-AI collaboration, users build a mental model of the AI system based on its reliability and how it presents its decision, e.g. its presentation of system confidence and an explanation of the output. Modern NLP systems are often uncalibrated, resulting in confidently incorrect predictions that undermine user trust. In order to build trustworthy AI, we must understand how user trust is developed and how it can be regained after potential trust-eroding events. We study the evolution of user trust in response to these trust-eroding events using a betting game. We find that even a few incorrect instances with inaccurate confidence estimates damage user trust and performance, with very slow recovery. We also show that this degradation in trust reduces the success of human-AI collaboration and that different types of miscalibration -- unconfidently correct and confidently incorrect -- have different negative effects on user trust. Our findings highlight the importance of calibration in user-facing AI applications and shed light on what aspects help users decide whether to trust the AI system.

• The paper demonstrates that miscalibrated confidence—both confidently incorrect and unconfidently correct predictions—significantly undermines user trust in AI systems.
• Experiments using a betting game reveal that even a few confidently incorrect predictions lead to persistent trust degradation and decreased collaborative performance.
• Predictive modeling with logistic regression and GRU networks effectively captures the temporal dynamics of user-AI interactions, emphasizing the need for robust calibration.

Understanding User Trust in AI Under Uncertainty

Introduction

The paper "A Diachronic Perspective on User Trust in AI under Uncertainty" presents a thorough investigation into the dynamics of user trust in AI systems, particularly how miscalibrated confidence signals can erode trust over time. This study is essential in understanding user-AI interactions, especially in high-stakes environments where AI decisions have significant implications. Through a series of experiments, the authors explore the factors influencing trust degradation and demonstrate the importance of accurate confidence calibration in fostering effective human-AI collaboration.

Figure 1: Diachronic view of a typical human-AI collaborative setting. At each timestep $t$, the user uses their prior mental model $\psi_t$ to accept or reject the AI system's answer $y_t$, supported by an additional message $m_t$ (AI's confidence), and updates their mental model of the AI system to $\psi_{t+1}$.

Miscalibration and its Effects on User Trust

The central hypothesis is that user trust in AI systems can be severely affected by miscalibration, where an AI's confidence does not align with its prediction accuracy. The research categorizes miscalibration into confidently incorrect (CI) and unconfidently correct (UC) predictions and examines their distinct impacts on user trust.

• Confidently Incorrect (CI) Effects: CI predictions result in users placing incorrect trust in AI outputs, which significantly damages the perceived reliability of the AI system. The study shows that even a small number of CI examples can have a lasting negative effect on user trust.
• Unconfidently Correct (UC) Effects: Conversely, UC predictions tend to have a lesser impact on trust degradation. Users are more forgiving of underconfident predictions that are ultimately correct, although they still contribute to a decline in collaboration performance.

  Figure 2: Possible correctness and confidence combinations of an AI system. CI and UC are miscalibrated, while the rest are calibrated.

Experimental Setup and Findings

The experiments involve a betting game where users interact with a simulated AI system that provides predictions with confidence levels. Users bet on the validity of the AI's predictions, serving as a proxy for trust. The study observes that:

• Miscalibrated examples lead to a significant decrease in user trust and performance.
• Trust recovery is slow after exposure to CI predictions, even when subsequent predictions are calibrated.
• Trust miscalibration impacts not only immediate interactions but can affect perceptions of unrelated prediction types.

  Figure 3: Pipeline for a single stimulus out of 60. The maximum payout for a bet is 10¢. Purple boxes show possible user actions.

Modeling and Implications

To better understand trust dynamics, various predictive models, including logistic regression and GRU networks, are developed to estimate user trust and predict user decisions based on interaction history and AI confidence levels. Recurrent models like GRUs capture the complex temporal dynamics more effectively than simple linear models.

The study's findings underscore the necessity for robust calibration methods in AI systems, emphasizing that even subtle miscalibrations can have profound effects on trust and collaborative efficiency.

Figure 4: Average user bet values (y-axis) and bet correctness with no intervention (control, top) and CI intervention (bottom).

Conclusion

This paper highlights the critical role of confidence calibration in AI systems, especially in high-stakes decision scenarios where user trust is integral to success. The results point towards the need for user-centric design approaches that consider trust dynamics, facilitating more reliable human-AI collaborations. Future research should focus on strategies for quickly regaining trust and explore diverse reward structures in real-world applications.

Continued exploration into trust dynamics will aid the development of AI systems that align more closely with human expectations, fostering safer and more effective integrations in various domains.