Why Language Models Hallucinate

Abstract: Like students facing hard exam questions, LLMs sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty. Such "hallucinations" persist even in state-of-the-art systems and undermine trust. We argue that LLMs hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty, and we analyze the statistical causes of hallucinations in the modern training pipeline. Hallucinations need not be mysterious -- they originate simply as errors in binary classification. If incorrect statements cannot be distinguished from facts, then hallucinations in pretrained LLMs will arise through natural statistical pressures. We then argue that hallucinations persist due to the way most evaluations are graded -- LLMs are optimized to be good test-takers, and guessing when uncertain improves test performance. This "epidemic" of penalizing uncertain responses can only be addressed through a socio-technical mitigation: modifying the scoring of existing benchmarks that are misaligned but dominate leaderboards, rather than introducing additional hallucination evaluations. This change may steer the field toward more trustworthy AI systems.

• The paper demonstrates that hallucinations are natural statistical errors, formalized as misclassifications in the 'Is-It-Valid' (IIV) binary framework.
• It categorizes error sources such as epistemic uncertainty, model misspecification, computational hardness, distribution shift, and data quality issues.
• The study proposes modifying evaluation protocols to reward calibrated uncertainty and abstention, thereby mitigating overconfident guessing.

Statistical and Socio-Technical Causes of Hallucination in LLMs

The paper "Why LLMs Hallucinate" (2509.04664) provides a rigorous statistical and socio-technical analysis of hallucinations in LLMs. The authors argue that hallucinations—plausible but incorrect outputs—are not mysterious artifacts but are instead a natural consequence of the objectives and evaluation protocols used in modern LLM training and deployment. The work formalizes the connection between generative errors in LLMs and misclassification in supervised learning, and it critiques the prevailing evaluation methodologies that inadvertently incentivize overconfident guessing over calibrated uncertainty.

Formalizing Hallucination as a Statistical Error

The central thesis is that hallucinations in LLMs are fundamentally errors in binary classification, specifically in distinguishing valid from invalid outputs. The authors introduce the "Is-It-Valid" (IIV) binary classification problem, where the model must classify outputs as valid or erroneous. They show that the generative error rate of a LLM is lower-bounded by twice the IIV misclassification rate, modulo calibration and class imbalance terms. This reduction is illustrated in (Figure 1).

Figure 1: Is-It-Valid requires learning to identify valid generations using labeled $\pm$ examples (left). Classifiers (dashed lines) may be accurate on certain concepts like spelling (top) but errors often arise due to poor models (middle) or arbitrary facts when there is no pattern in the data (bottom).

This formalism demonstrates that even with error-free training data, the statistical objectives optimized during pretraining (e.g., cross-entropy minimization) will inevitably lead to some generative errors. The analysis is agnostic to model architecture and applies to both next-token prediction and general density estimation.

Taxonomy of Error Sources

The paper categorizes the statistical sources of hallucination:

• Epistemic Uncertainty (Arbitrary Facts): When facts are unpatterned and appear only once in the training data (e.g., rare birthdays), the model cannot generalize, and the hallucination rate is at least the singleton rate in the data.
• Model Misspecification: If the model class cannot represent the true data distribution (e.g., trigram models for long-range dependencies), high error rates are inevitable.
• Computational Hardness: For problems that are computationally intractable (e.g., cryptographic decryption), no efficient model can avoid errors.
• Distribution Shift: Out-of-distribution prompts induce errors due to mismatch between training and test distributions.
• GIGO (Garbage In, Garbage Out): Errors in the training data are replicated by the model.

The analysis is extended to prompted settings, where the model must generate valid responses conditioned on a prompt, and the same lower bounds on error rates apply.

Calibration and Its Limits

The authors show that calibration—matching predicted probabilities to empirical frequencies—is a natural outcome of cross-entropy minimization in pretraining. However, perfect calibration does not preclude hallucinations; in fact, a perfectly calibrated model will still hallucinate on unlearnable or rare facts. Empirical evidence from GPT-4 calibration histograms (Figure 2) supports this claim: pretraining yields well-calibrated models, but post-training (e.g., via RLHF) can distort calibration.

Figure 2: GPT-4 calibration histograms before (left) and after (right) reinforcement learning. The pretrained model is well calibrated.

Socio-Technical Drivers: Evaluation and Benchmarking

A key contribution is the identification of evaluation protocols as a major driver of persistent hallucinations. Most mainstream benchmarks use binary grading (correct/incorrect), awarding no credit for abstaining or expressing uncertainty (e.g., "I don't know"). This incentivizes models to guess when uncertain, as abstention is strictly suboptimal under these metrics. The authors formalize this with a simple observation: under binary grading, the optimal strategy is always to guess, never to abstain.

The paper surveys popular benchmarks and finds that nearly all penalize uncertainty, reinforcing overconfident guessing. Even hallucination-specific evaluations are insufficient if the primary benchmarks remain misaligned.

Mitigation: Realigning Evaluation Protocols

The authors propose a socio-technical mitigation: modify existing benchmarks to explicitly reward calibrated uncertainty. This can be achieved by:

• Explicit Confidence Targets: Incorporate confidence thresholds and penalties for incorrect answers, as in some standardized human exams. For example, only answer if confidence exceeds $t$, with penalties for incorrect answers and zero credit for abstention.
• Behavioral Calibration: Evaluate models on their ability to abstain or hedge when uncertain, not just on accuracy.

Such modifications would realign incentives, encouraging models to express uncertainty appropriately and reducing the prevalence of hallucinations.

Implications and Future Directions

The theoretical results imply that hallucinations are an unavoidable consequence of statistical learning under current objectives and data regimes, especially for rare or unpatterned facts. However, the persistence and overconfidence of hallucinations are exacerbated by evaluation protocols that penalize uncertainty. The work suggests that meaningful progress on trustworthy AI requires not just technical advances in modeling, but also changes in how models are evaluated and benchmarked.

Future research directions include:

• Developing and adopting evaluation protocols that reward calibrated uncertainty and abstention.
• Exploring richer forms of uncertainty expression beyond "I don't know," such as hedging and clarification requests.
• Investigating the interplay between model architecture, training objectives, and behavioral calibration.
• Extending the statistical framework to open-ended and multi-fact generations, and to settings with latent or ambiguous context.

Conclusion

"Why LLMs Hallucinate" provides a principled statistical and socio-technical account of hallucinations in LLMs. The inevitability of generative errors is rooted in the fundamental limits of statistical learning, while the persistence of overconfident hallucinations is reinforced by misaligned evaluation protocols. Addressing hallucination thus requires both technical and institutional changes, particularly in how the field defines and rewards trustworthy model behavior.