Incoherent Probability Judgments in Large Language Models

Abstract: Autoregressive LLMs trained for next-word prediction have demonstrated remarkable proficiency at producing coherent text. But are they equally adept at forming coherent probability judgments? We use probabilistic identities and repeated judgments to assess the coherence of probability judgments made by LLMs. Our results show that the judgments produced by these models are often incoherent, displaying human-like systematic deviations from the rules of probability theory. Moreover, when prompted to judge the same event, the mean-variance relationship of probability judgments produced by LLMs shows an inverted-U-shaped like that seen in humans. We propose that these deviations from rationality can be explained by linking autoregressive LLMs to implicit Bayesian inference and drawing parallels with the Bayesian Sampler model of human probability judgments.

• The paper demonstrates that LLMs produce incoherent probabilistic judgments deviating from theoretical expectations in probability identities.
• It employs controlled experiments on weather and political events with varied temperature settings to highlight biases across GPT and LLaMA models.
• Findings indicate that larger models reduce variance and incoherence, linking these deviations to Bayesian sampling processes in human-like reasoning.

Incoherent Probability Judgments in LLMs

This paper addresses the coherence of probability judgments expressed by Autoregressive LLMs, scrutinizing whether such models, known for producing coherent text, maintain coherence when tasked with probabilistic reasoning.

Evaluating Coherence in LLMs

The study focuses on the connection between autoregressive LLM outputs and human-like probability judgments, specifically examining common systematic deviations from probability theory.

Methods

Four LLMs—GPT-3.5-turbo, GPT-4, LLaMA-2-7b, and LLaMA-2-70b—were evaluated using probabilistic identities that ideally should equate to zero, representing probability coherence. The LLMs were prompted to assign probabilities to event pairs related to weather and politics, using a uniform framing of the queries to ensure consistency. Different temperature settings (0 and 1) were utilized to explore model behavior under varying levels of stochasticity, and the coherence of responses was gauged through multiple repetitions of identical prompts at temperature 1.

Results

The results revealed consistent biases among LLMs, characterized by systematic deviations in the probabilistic identities comparable to those observed in human reasoning.

Figure 1: Bias and variability in human probability judgments as revealed by (left) probabilistic identities and (right) mean-variance relationship. Error bars are 95% CI.

LLMs' probabilistic identities did not equate to zero, indicating incoherence. These deviations were influenced by the imbalance of positive and negative terms within the identities, mirroring patterns seen in humans.

Figure 2: Probabilistic identities based on LLM responses. For coherent judgments, identities should be zero.

Mean-variance relationships in repeated probability judgments also exhibited inverted-U shapes. Variants with more parameters demonstrated decreased deviation and lowered variance, suggesting that increased model size enhances coherence but fails to achieve perfection.

Figure 3: The relationship between mean and variance in repeated probability judgments shows an inverted-U shape.

Theoretical Implications

Human-Like Deviations

The study relates the incoherent judgments of LLMs to patterns in human probabilistic reasoning, explained by the Probability Theory plus Noise (PT+N) model and the Bayesian Sampler model. Both these models propose intrinsic processes for generating probability samples, explaining deviations from coherence through various factors like additional noise or the influence of priors.

Bayesian Framework for LLMs

The paper proposes a Bayesian interpretation of LLM judgments: autoregressive processes can be linked to implicit Bayesian inference, where the LLM’s conditional probability predictions are viewed as Bayesian updates on prior distributions. The paper discusses how autoregressive training aligns with Bayesian mechanisms, suggesting that LLMs' deviations have parallels to Bayesian sampling in humans.

Future Directions

The research implies a potential strategy for enhancing the reliability of AI probability outputs: adjusting incoherence rather than just calibrating against true frequencies. The coherence-accuracy relationship in bounded rational agents supports this recalibrative approach, opening avenues for refining AI's probabilistic outputs to be more reliable for practical applications.

Conclusion

The examination of LLM coherence when making probabilistic judgments reveals similarities with human cognitive biases, facilitated by autoregressive training processes. This understanding bridges neural network methodologies with Bayesian reasoning models, contributing to the broader comprehension of both human-like and artificial cognitive processes.