Explainable Human-AI Interaction: A Planning Perspective

Abstract: From its inception, AI has had a rather ambivalent relationship with humans -- swinging between their augmentation and replacement. Now, as AI technologies enter our everyday lives at an ever increasing pace, there is a greater need for AI systems to work synergistically with humans. One critical requirement for such synergistic human-AI interaction is that the AI systems be explainable to the humans in the loop. To do this effectively, AI agents need to go beyond planning with their own models of the world, and take into account the mental model of the human in the loop. Drawing from several years of research in our lab, we will discuss how the AI agent can use these mental models to either conform to human expectations, or change those expectations through explanatory communication. While the main focus of the book is on cooperative scenarios, we will point out how the same mental models can be used for obfuscation and deception. Although the book is primarily driven by our own research in these areas, in every chapter, we will provide ample connections to relevant research from other groups.

• The paper introduces a planning framework that enables AI to generate minimal, complete explanations aligning with human mental models.
• It details methodologies for reconciling AI plans with human expectations through explicability, legibility, and predictability.
• The study examines trade-offs between explanation detail and computational overhead, offering practical insights for high-stakes applications.

Explainable Human-AI Interaction: A Planning Perspective

The discussed paper, "Explainable Human-AI Interaction: A Planning Perspective" by Sarath Sreedharan, Anagha Kulkarni, and Subbarao Kambhampati, from Arizona State University, centers around the growing necessity for AI systems to interact transparently and cooperatively with humans. This necessity roots from the fact that as AI systems become increasingly pervasive in everyday applications, their decision-making processes must be comprehensible to human stakeholders. The paper explores the aspects and methodologies of creating AI systems that can explain their actions, decisions, and plans to humans effectively, emphasizing cooperative human-AI interactions, obfuscation, and deception scenarios.

Overview

The paper systematically explores the field of explainability in AI through various dimensions. It outlines that unlike conventional AI systems designed to function remotely or adversarially compared to humans (e.g., AI in games like Chess or Go), the goal is to develop systems that can actively engage and build trust with humans. This can be particularly critical in high-stakes domains such as healthcare or criminal justice where AI decisions significantly impact human lives.

Dimensions of Explainable AI Systems

The paper identifies multiple dimensions along which explainable AI systems can be evaluated:

1. Explicability: The extent to which an AI’s actions align with human expectations.
2. Legibility: The ability of an AI system to signal its goals or plans through its actions.
3. Predictability: Ensuring that an AI's actions can be anticipated by human observers.

Each dimension highlights a different aspect of interaction between humans and AI, focusing on reducing the cognitive load for humans to understand AI behavior and improving the overall human-AI teaming efficiency.

Explanation Framework

A crucial aspect of the discussion focuses on model reconciliation. Instead of modifying the AI's plan to meet human expectations (as in explicable planning), the AI provides explanations to humans, thereby altering their understanding and expectations. This is achieved by communicating relevant aspects of the AI’s model that the human may not be aware of. The goal is to generate minimally complete explanations (MCE) that are concise but sufficient to make the given plans understandable and seem optimal to humans.

Approximate Explanations

The paper addresses how explanations and the necessary model information can be adjusted to account for the human observer's limited inferential capabilities. It discusses the trade-offs involved in finding the balance between the size of the explanation and the computational overhead involved in generating them.

Acquisition of Mental Models for Explanations

Detailed methods are proposed to handle scenarios where an AI does not possess an accurate model of the human's mental state initially. These methods include:

1. Incomplete Models: Addressing situations where partial information about the human’s mental model is known.
2. Model-Free Approaches: Learning approximate models from human feedback.
3. Prototypical Models: Assuming simpler representations of human mental models for ease of explanation.
4. Annotation and Robustness: Employing annotated models to gauge the robustness of explanations across possible human mental models.

Implications and Future Directions

The practical implications of this research are immense. In real-world scenarios such as urban search and rescue, medical decision support systems, and autonomous driving, the capability of an AI system to explain its actions fosters transparency and trust, crucial for human acceptance and collaboration.

Furthermore, the paper explores environmental design as a means to facilitate explicable behavior in repetitive tasks, emphasizing the importance of a synergistic relationship between environment modification and human-AI interaction strategies.

Theoretical and Practical Contributions

On a theoretical front, this research bridges the gap between AI planning and human-computer interaction (HCI), proposing comprehensive frameworks that formalize explainable behaviors and explanation generation mechanisms. On a practical level, the described methods and algorithms can be employed to develop more transparent and trustable AI systems.

Conclusion

The pursuit of explainable AI systems is critical for the integration of AI into domains where human collaboration is essential. This paper provides a detailed roadmap for achieving this through robust planning, explanation strategies, and mental model reconciliation, catering to both explanatory needs and computational efficiency. Future developments may focus on integrating more complex human cognitive models, allowing for even more nuanced and effective explanations.

By instilling the ability to explain, this research aims to foster AI systems that not only perform optimally but do so in a manner that engenders human trust and collaboration, marking significant strides towards human-aware AI systems.