Human Agency, Causality, and the Human Computer Interface in High-Stakes Artificial Intelligence

Abstract: Current discourse on AI ethics, dominated by "trustworthy" and "responsible" AI, overlooks a more fundamental human-computer interaction (HCI) crisis: the erosion of human agency. This paper argues that the primary challenge of high-stakes AI systems is not trust, but the preservation of human causal control. We posit that "bad AI" will function as "bad UI," a metaphor for catastrophic interface failures that misrepresent system state and lead to human error. Applying Marshall McLuhan's media theory, AI can be framed as a technology of "augmentation" that simultaneously "amputates" the user's direct perception of causality. This places the interface as the critical locus where a "double uncertainty"--that of the human user and that of the probabilistic model--must be mediated. We critique current Explainable AI (XAI) for its correlational focus and failure to represent uncertainty. We conclude by proposing a rigorous, nested Causal-Agency Framework (CAF) that integrates causal models, uncertainty quantification, and human-centered evaluation to restore agency at the interface.

• The paper demonstrates that embedding causal reasoning and uncertainty quantification within AI interfaces is essential to restore actionable human agency.
• It introduces a Causal-Agency Framework that integrates Structural Causal Models with a human-centered interface for direct intervention and feedback control.
• The study redefines evaluation metrics from trust and readability to outcome-centric performance, demanding richer data and enhanced causal literacy.

Human Agency, Causality, and the Human-Computer Interface in High-Stakes AI

Introduction

This paper, "Human Agency, Causality, and the Human Computer Interface in High-Stakes Artificial Intelligence" (2604.12793), critically situates the dominant discourses in AI ethics—e.g., "trustworthy" and "responsible" AI—as insufficient for managing the actual risks AI systems pose in high-stakes contexts. Instead, it posits that the most urgent challenge is the preservation and operationalization of human agency at the locus of the human-computer interface (HCI). The central argument is that "bad AI" will increasingly function as "bad UI," disabling users from acting as causal agents and inducing catastrophic outcomes through misrepresented system states and ambiguous affordances. The paper advances this thesis by developing a theoretically and technically integrated Causal-Agency Framework (CAF), which extends beyond the explainability paradigm to embed causality and uncertainty quantification as first-class computational primitives of high-stakes AI systems.

Historical and Theoretical Motivation

The analysis is grounded in extensive HCI and human factors scholarship, using case studies such as the Therac-25, aviation automation failures (Air France 447), nuclear system incidents (Three Mile Island), defense automation errors (USS Vincennes), and EHR-induced medical errors. These incidents are unified by the failure of interfaces to represent causal linkages between user intention and system state, resulting not in mere "human error" but rather in a breakdown of actionable feedback control. The paper adopts and extends McLuhan's conception of technological "augmentation" and "amputation," asserting that AI—by mediating and computationally abstracting the causal chain—attacks not only memory or calculation, but the user's direct perception of causality itself. This leads to a "double uncertainty": model uncertainty (aleatoric and epistemic) and user uncertainty, which together degrade the user's status as an agentic operator.

Critique of Explainable AI and Its Limits

The paper offers a rigorous critique of the explainability agenda, demonstrating that current explainable AI (XAI) methods (e.g., LIME, SHAP) are fundamentally correlational and do not supply the necessary counterfactual or interventional knowledge users require for causal reasoning. It underscores that most human users, when asking "why," are engaging in counterfactual inquiry—a determination of necessary interventions to attain or avert outcomes—not a request for an enumeration of features with high attribution weights. It further highlights that XAI explanations, as presently operationalized, fail to systematically represent and propagate model uncertainty, often communicating brittle or unearned confidence and abetting automation bias.

The paper forcefully claims that the core design metrics for success in high-stakes AI are not user "trust" or subjective satisfaction, but the restoration of measurable, actionable human agency—where the human operator can intentionally intervene and observe the causal effect of actions, informed by an epistemic account of risk.

The Causal-Agency Framework (CAF)

To address these challenges, the paper introduces the Causal-Agency Framework (CAF), which conceptualizes high-stakes AI system architecture as a nested, multilayered structure comprising:

1. Causal content Uncertainty Quantification (CUQ) Engine: The technical substrate, built around Structural Causal Models (SCMs) and rigorous uncertainty quantification, moving from associational $P(Y|X)$ to interventional $P(Y|do(X))$ queries, and distinguishing between epistemic and aleatoric uncertainty.
2. Explanation content Intervention (E{content}I) Module: A translational/mediational layer that produces actionable, human-meaningful counterfactuals and risk statements from the technical backend, adaptively tailoring explanations to context and user intent.
3. Human-Centered Agency Interface (HCAI): The user-facing "cockpit" (not dashboard) that exposes causal affordances, allows for direct intervention, supports overrides and constraint imposition, and is driven by closed-loop evaluation of the joint human-AI system's real-world performance.

This separation of concerns (technical, translational, interactional) is posited as essential for both scientific progress and responsible system validation.

Practical and Theoretical Implications

The implications of the CAF are manifold.

• Methodological Shift: The architecture rejects post hoc explainability applied to black boxes, advancing "interpretability by design" where causality and uncertainty are embedded primitives.
• Evaluation Redefined: Metrics must move from user trust and readability to outcome-centric and closed-loop joint performance, emphasizing controllability and scrutability.
• Structural Agency as a Prerequisite: Agency is recast not as a byproduct of psychological "framing" or trust manipulation, but as a direct function of interface affordances and epistemic access to causal structure.
• Practical Data Imperative: The CAF exposes limitations of purely observational "big data" for causal recourse; implementation will require richer and more interventionally annotated datasets, and possibly new technical approaches for confounder detection.
• Educational Consequence: Practitioners and domain experts must develop further causal and uncertainty literacy to operationalize the system's affordances, transcending basic digital or statistical fluency.

A strong empirical claim is advanced: citing research showing that human-AI teams often fail to outperform AI alone, the paper attributes this to an interface crisis wherein "double uncertainty" obviates the intended advantages of collaboration, and poorly designed interfaces create more errors, not fewer.

Future Directions

By presenting a formal framework for nesting causality and uncertainty quantification through to actionable interface affordances, the paper sets an agenda for future research. This encompasses new causal discovery algorithms, uncertainty propagation protocols, participatory HCI methodologies, and the development of new closed-loop evaluation schemes tailored for high-stakes settings. There is also a data and institutional challenge to recalibrate both data collection and operator training strategies to accommodate causal and agentic requirements.

Conclusion

The paper offers a methodological and architectural reframing of high-stakes AI system design, arguing for an explicit shift from trust-oriented paradigms to agency-centric technical architectures. The CAF provides a meta-structural template for aligning the epistemic affordances of AI with the situated needs of human operators. As AI systems are given greater operational control in high-consequence domains, the ability to maintain human causal influence is made not only a desideratum for safety, but the minimal baseline for ethical alignment. The proposed research agenda has broad implications for AI, HCI, and domain-specific interface design in critical infrastructure and health informatics.

Reference:

"Human Agency, Causality, and the Human Computer Interface in High-Stakes Artificial Intelligence" (2604.12793)