Hallucinations Undermine Trust; Metacognition is a Way Forward

Abstract: Despite significant strides in factual reliability, errors -- often termed hallucinations -- remain a major concern for generative AI, especially as LLMs are increasingly expected to be helpful in more complex or nuanced setups. Yet even in the simplest setting -- factoid question-answering with clear ground truth-frontier models without external tools continue to hallucinate. We argue that most factuality gains in this domain have come from expanding the model's knowledge boundary (encoding more facts) rather than improving awareness of that boundary (distinguishing known from unknown). We conjecture that the latter is inherently difficult: models may lack the discriminative power to perfectly separate truths from errors, creating an unavoidable tradeoff between eliminating hallucinations and preserving utility. This tradeoff dissolves under a different framing. If we understand hallucinations as confident errors -- incorrect information delivered without appropriate qualification -- a third path emerges beyond the answer-or-abstain dichotomy: expressing uncertainty. We propose faithful uncertainty: aligning linguistic uncertainty with intrinsic uncertainty. This is one facet of metacognition -- the ability to be aware of one's own uncertainty and to act on it. For direct interaction, acting on uncertainty means communicating it honestly; for agentic systems, it becomes the control layer governing when to search and what to trust. Metacognition is thus essential for LLMs to be both trustworthy and capable; we conclude by highlighting open problems for progress towards this objective.

• The paper identifies a trade-off where reducing hallucinations via stricter confidence thresholds leads to a significant loss of correct answers, illustrating a costly utility tax.
• It introduces the concept of 'faithful uncertainty' to align linguistic expressions of uncertainty with intrinsic confidence, thereby improving factual reliability.
• Empirical results using AUROC metrics reveal the discriminative gap in LLMs, underscoring the need for enhanced metacognitive control in agentic systems.

Metacognition for Hallucination Mitigation in LLMs: Addressing the Utility-Factuality Trade-off

Introduction and Motivation

LLMs are increasingly deployed in knowledge-intensive settings, yet their persistent generation of factual errors—commonly characterized as hallucinations—undermines reliability and user trust. Despite observable improvements in factual accuracy with larger model scales and improved training, even frontier models exhibit hallucinations under factoid QA with clear ground truth. This work identifies that recent progress has predominantly improved models’ knowledge boundaries (i.e., encoded world facts) rather than their awareness of those boundaries—the discriminative capacity to distinguish what is known from unknown. The paper introduces the thesis that enhancing the latter is inherently difficult, leading to a trade-off where aggressively suppressing hallucinations comes at a significant "utility tax": many correct answers are lost due to over-abstention.

Theoretical and Empirical Analysis of the Discriminative Gap

The central technical contribution is the formalization and empirical substantiation of the discriminative gap. The paper differentiates calibration (confidence statements are accurate on average) from discrimination (confidence effectively separates correct from incorrect outputs at the instance level), showing that standard calibration metrics are insufficient. Well-calibrated LLMs demonstrate high error overlap across confidence intervals: reducing the hallucination rate from 25% to 5% under best-available internal signals requires discarding over half of correct answers.

Figure 1: Strong calibration in LLMs does not entail discrimination—the empirical trade-off demonstrates that utility is sacrificed aggressively as stricter factuality targets are imposed, due to significant overlap in confidence distributions for correct and incorrect responses.

A review of empirical AUROC scores from multiple benchmarks (e.g., SimpleQA Verified, biography generation, medical QA) indicates that current LLMs routinely achieve only 0.70–0.85 discrimination, far from the near-perfect $\geq$0.95 required to avoid a prohibitive utility tax. This landscape is illustrated using empirical benchmark data, where models must navigate between maximizing coverage (but permitting more hallucinations) or severely curtailing utility to improve reliability.

Figure 2: Models cluster along a coverage-factuality diagonal, with no models occupying the region associated with both maximal utility and near-zero hallucination rates, directly visualizing the discriminative gap.

Reframing Hallucination: Faithful Uncertainty and Metacognition

The work’s main proposal is to move beyond the binary answer-or-abstain paradigm by reframing hallucinations specifically as confident errors: incorrect statements delivered without epistemic qualification. This motivates faithful uncertainty as an actionable objective for LLMs—the alignment between expressed (linguistic) uncertainty and true model (intrinsic) uncertainty. If a model reliably signals its lack of confidence, errors communicated with appropriate hedging do not constitute hallucinations in this reframing; they are instead presented as hypotheses, preserving utility without misleading users.

Figure 3: The faithful uncertainty paradigm emerges as a third axis, enabling LLMs to avoid the utility-factuality trade-off by honestly expressing internal uncertainty—attenuating the harmful impact of confident hallucinations.

Faithful uncertainty is defined at the instance level by the correspondence between decisiveness in language and intrinsic confidence (established via the consistency of responses under repeated sampling). A formal measurement framework is referenced, including the conditional Mean Faithful Generation (cMFG) metric.

Metacognition as Control: Implications for Agentic LLMs

The utility of faithful uncertainty extends into agentic LLMs—those that use external tools such as search APIs or code execution. Here, metacognition functions as the control layer: the model must decide when retrieval is required, when to accept retrieved evidence, and how to arbitrate between parametric and tool-augmented information. Without accurate self-estimation of knowledge gaps, agents degenerate into inefficient or hazardous behaviors, such as making unnecessary tool calls or accepting unreliable external facts uncritically.

Figure 4: Metacognitive awareness enables LLM-based agents to dynamically and efficiently control tool invocation and resolve conflicts between parametric knowledge and retrieved evidence.

Empirically, current agentic systems often display tool overuse for routine queries and fail to modulate reliance on internal versus external information, traceable to an absence of robust metacognitive introspection and regulation.

Methodological Challenges and Recommendations

The development of metacognitive LLMs introduces novel challenges:

• Bootstrapping Paradox: Hedging data for fine-tuning is static, whereas the model’s actual knowledge boundary shifts during continued training, risking outdated or misaligned uncertainty signals.
• Preservation across Post-training: Tuning for safety or instruction-following typically degrades well-calibrated uncertainty representations evident in base models, suggesting a need for "uncertainty-preserving" alignment methods.
• Linguistic Attribution: Faithful uncertainty is not reducible to scalar scores—a model must attribute uncertainty to epistemic, aleatoric, or normative causes and map these onto precise linguistic markers.
• Rigorous Causal Evaluation: Superficial hedging strategies (e.g., always hedging on rare names) must be distinguished from genuinely introspective metacognitive abilities; mechanistic probes and adversarial tasks are recommended for evaluation.

  Figure 5: Summary of key recommendations and open problems for advancing metacognition and faithful uncertainty in LLM research.

Evaluating hallucination mitigation methods also requires visualizing utility-error trade-offs holistically, ensuring claimed improvements yield better factuality/utility frontiers rather than mere threshold shifts or performance regression on other axes.

Implications and Future Outlook

The central assertion is that faithful uncertainty is theoretically attainable—it requires only internal alignment, not omniscient external calibration. This reframing does not diminish the importance of knowledge expansion but complements it: factuality gains push the boundary outward, while faithful uncertainty ensures reliable communication across whatever boundary remains. As LLMs become the substrate for increasingly autonomous agents, robust metacognitive capabilities become prerequisites for safe and efficient deployment.

The work identifies tractable research headroom in this space, citing that current LLMs (even those displaying strong calibration) are poorly aligned at the instance level and often fail to convey true uncertainty linguistically. Developments in introspective prompting, fine-tuning on epistemic expressions, and mechanistic confidence probe extraction are highlighted as promising avenues for progress.

Conclusion

This paper makes a rigorous case that hallucinations in LLMs, particularly confident errors, cannot be eradicated solely by scaling knowledge. Instead, the discrimination gap imposes hard constraints on the balance between reliability and utility, with existing approaches either trading off coverage or perpetuating trust-eroding hallucinations. By centering faithful uncertainty—the metacognitive alignment of linguistic and intrinsic confidence—as an actionable objective, the work describes a pathway that preserves both trust and capability, especially as LLMs become agents capable of using external tools. The conceptual and empirical toolkit provided here defines the agenda for the next phase of research in factuality, reliability, and AI safety.

---

Reference: "Hallucinations Undermine Trust; Metacognition is a Way Forward" (2605.01428)