Understanding Understanding: A Pragmatic Framework Motivated by Large Language Models

Abstract: Motivated by the rapid ascent of LLMs and debates about the extent to which they possess human-level qualities, we propose a framework for testing whether any agent (be it a machine or a human) understands a subject matter. In Turing-test fashion, the framework is based solely on the agent's performance, and specifically on how well it answers questions. Elements of the framework include circumscribing the set of questions (the "scope of understanding"), requiring general competence ("passing grade"), avoiding "ridiculous answers", but still allowing wrong and "I don't know" answers to some questions. Reaching certainty about these conditions requires exhaustive testing of the questions which is impossible for nontrivial scopes, but we show how high confidence can be achieved via random sampling and the application of probabilistic confidence bounds. We also show that accompanying answers with explanations can improve the sample complexity required to achieve acceptable bounds, because an explanation of an answer implies the ability to answer many similar questions. According to our framework, current LLMs cannot be said to understand nontrivial domains, but as the framework provides a practical recipe for testing understanding, it thus also constitutes a tool for building AI agents that do understand.

• The paper introduces a framework that evaluates AI understanding by measuring average competence and ensuring the rarity of ridiculous answers.
• The paper employs probabilistic testing using concentration inequalities to derive high-confidence performance metrics from empirical samples.
• The paper demonstrates that incorporating explanation procedures can reduce sampling inefficiencies and guide improvements in AI reliability.

A Pragmatic Framework for Evaluating Understanding in LLMs

Understanding whether LLMs truly understand their subject matter is a critical question in the advancement of AI. Kevin Leyton-Brown and Yoav Shoham propose a rigorous framework to assess the understanding of any agent, human or machine, based solely on the agent's performance in answering questions. Inspired by the Turing Test, this framework defines understanding within a given domain through two primary criteria: average competence and avoidance of ridiculous answers.

Definition and Criteria of Understanding

The framework introduces a mathematical definition of understanding that circumvents vague and ill-defined concepts traditionally associated with the topic. It proposes a specific scope of understanding defined by a set of questions with a known distribution. The framework evaluates understanding based on:

1. Overall Passing Grade (PG): A threshold ensuring the average score across all questions exceeds a high predefined value.
2. Global Ridiculousness Threshold (RID): Ensuring that the probability of providing a ridiculous answer is negligibly small.

Procedural and Probabilistic Testing

Testing an agent's understanding through exhaustive questioning in nontrivial domains is infeasible. Instead, Leyton-Brown and Shoham suggest a probabilistic approach using random sampling and concentration inequalities. The proposed testing procedure uses empirical samples to draw high-confidence conclusions about the agent's competence and propensity to avoid ridiculous answers. Key insights include:

• The number of samples required for high-confidence testing can be substantial, potentially reaching thousands.
• Probabilistic guarantees are employed to ensure reliability, leveraging bounds derived from the Chernoff inequality for tighter confidence intervals compared to the Hoeffding bound when dealing with probabilities close to zero or one.

Impact of Explanations

The authors recognize that the inefficiency in sampling can be mitigated by explanations accompanying answers. Explanations demonstrate broader principles and justify answers, potentially covering multiple related questions. Formally, explanations are modeled through procedures applicable to sets of questions, which can significantly reduce the number of required samples:

• Trusted procedures, when reliably applied, extend coverage beyond individual answers.
• Empirical observations of procedures' usage further refine confidence bounds, albeit if the procedures' application is uncertain, additional sampling is required to maintain the same confidence levels.

Practical and Theoretical Implications

The proposed framework carries significant implications for both the evaluation and development of AI systems:

• Evaluation: The framework provides a robust method for assessing AI systems' understanding, revealing shortcomings such as occasional ridiculousness and lack of reliable explanations or admissions of ignorance.
• Development: Guiding AI development towards enhanced reliability, transparency, and the ability to provide explanations. These attributes are crucial for deploying AI in critical applications and ensuring user trust.

Overall, the framework's application confirms that current LLMs do not meet the stringent criteria for understanding in nontrivial domains. This finding underscores the necessity for continued research and development to address fundamental limitations in AI systems.

Future Directions

The framework opens several avenues for future research:

• Dynamic Understanding: Exploring how understanding can evolve through interactions and iterative learning processes.
• Scope Adaptation: Investigating mechanisms for dynamically expanding the scope of understanding based on uncovering new relevant questions during testing.
• Integration with Neuro-Symbolic Methods: Potentially combining data-driven approaches of LLMs with symbolically clear methods to align AI systems more closely with the framework's criteria.

In conclusion, by rigorously defining, mathematically formalizing, and probabilistically validating the concept of understanding, Leyton-Brown and Shoham's framework provides a solid foundation for advancing both theoretical understanding and practical development of AI systems.