Hallucination is Inevitable: An Innate Limitation of Large Language Models

Abstract: Hallucination has been widely recognized to be a significant drawback for LLMs. There have been many works that attempt to reduce the extent of hallucination. These efforts have mostly been empirical so far, which cannot answer the fundamental question whether it can be completely eliminated. In this paper, we formalize the problem and show that it is impossible to eliminate hallucination in LLMs. Specifically, we define a formal world where hallucination is defined as inconsistencies between a computable LLM and a computable ground truth function. By employing results from learning theory, we show that LLMs cannot learn all the computable functions and will therefore inevitably hallucinate if used as general problem solvers. Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs. Furthermore, for real world LLMs constrained by provable time complexity, we describe the hallucination-prone tasks and empirically validate our claims. Finally, using the formal world framework, we discuss the possible mechanisms and efficacies of existing hallucination mitigators as well as the practical implications on the safe deployment of LLMs.

• The paper proves that LLMs inevitably hallucinate as a fundamental consequence of computational complexity and learning constraints.
• It shows through rigorous proofs and empirical examples that even state-of-the-art models like Llama2 and GPT variants fail to model all computable functions accurately.
• The study discusses mitigation strategies such as structured prompts and external knowledge integration while emphasizing the need for safety safeguards in critical applications.

Hallucination is Inevitable: An Innate Limitation of LLMs

This paper explores the fundamental limitations of LLMs concerning their tendency to hallucinate, a situation where these models generate information that appears plausible but is actually incorrect or nonsensical. It posits that hallucination is an unavoidable consequence of how LLMs are implemented, rather than just an empirical artifact, using formal definitions and theoretical tools from learning theory to substantiate this claim.

Problem Definition and Theoretical Background

The paper begins by formally defining hallucination in LLMs. Hallucination is characterized by the inconsistency between the model's output and a theoretical ground truth function, a computable function that provides the correct completion of any input string. The authors argue that due to the inherent limitations in what an LLM can learn—constrained by computational complexity and learning theory—there exists no LLM capable of modeling all computable functions without hallucinating.

Inevitable Hallucination of LLMs

Through a series of theoretical results, it is demonstrated that:

1. Provability and Complexity Constraints: For LLMs proved by a computable algorithm (termed $P$-proved LLMs), hallucination is inevitable when faced with any computable ground truth function. This result is achieved via the diagonalization technique, indicating that any enumerated set of total computable functions for a given complexity level (e.g., polynomial time) excludes some ground truth functions that LLMs fail to model.
2. Enumerability and Learning Limitations: The paper extends the analysis to highlight that LLMs, when regarded as a part of any computably enumerable set, will hallucinate on an infinite subset of inputs, thereby demonstrating that hallucination cannot be avoided even with advances in LLM architectures.
3. General Case for All LLMs: Finally, it argues that any computable LLM is fated to hallucinate on infinite queries for any computable but unlearnable ground truth function. This fundamental result serves as a precise answer to the overarching question of whether complete elimination of hallucination is possible: it is not.

Empirical Validation

Empirical studies are conducted to illustrate these theoretical insights, focusing particularly on problems known to be difficult for polynomial-time models, such as exhaustive string listing and determining the linear order of strings. State-of-the-art LLMs, including Llama2 and GPT variants, demonstrate their limitations in handling these tasks, emphasizing the practical implications of the theoretical results.

Discussion on Mitigation Strategies

The paper discusses existing and potential methods for alleviating hallucination:

• Model and Data Scale: Increasing model size or dataset scale theoretically enhances capacity but does not address the fundamental limitations against certain function types.
• Prompts and In-Context Learning: Using structured prompts and reasoning chains can guide LLMs toward better performance on specific tasks but cannot universally eliminate hallucination.
• External Knowledge Systems: Integrating databases and retrieval-augmented generation (RAG) improves performance but introduces complexity in real-world deployment.

Practical Implications and Conclusion

This research underscores the necessity of recognizing the safety and reliability boundaries of LLMs. It suggests adopting safeguard frameworks when deploying LLMs in critical applications to circumvent the risks associated with inevitable hallucinations. While acknowledging the impactful capabilities of LLMs, it cautions against over-reliance on these models for decisions demanding high precision. Conclusively, the findings advocate for continued research into the mechanisms and preprocessing techniques that might better anticipate and reduce hallucination, but acknowledge that completely eradicating it is beyond current computational paradigms.