From Loops to Oops: Fallback Behaviors of Language Models Under Uncertainty

Abstract: LLMs often exhibit undesirable behaviors, such as hallucinations and sequence repetitions. We propose to view these behaviors as fallbacks that models exhibit under epistemic uncertainty, and investigate the connection between them. We categorize fallback behaviors - sequence repetitions, degenerate text, and hallucinations - and extensively analyze them in models from the same family that differ by the amount of pretraining tokens, parameter count, or the inclusion of instruction-following training. Our experiments reveal a clear and consistent ordering of fallback behaviors, across all these axes: the more advanced an LLM is (i.e., trained on more tokens, has more parameters, or instruction-tuned), its fallback behavior shifts from sequence repetitions, to degenerate text, and then to hallucinations. Moreover, the same ordering is observed during the generation of a single sequence, even for the best-performing models; as uncertainty increases, models shift from generating hallucinations to producing degenerate text and finally sequence repetitions. Lastly, we demonstrate that while common decoding techniques, such as random sampling, alleviate unwanted behaviors like sequence repetitions, they increase harder-to-detect hallucinations.

• The paper reveals that fallback behaviors, such as sequence repetitions and hallucinations, shift as model scale and training tokens increase.
• The paper demonstrates that instruction tuning improves alignment with human expectations but also heightens hallucination rates.
• The paper highlights the need for improved decoding methods to balance avoiding repetitive outputs with minimizing dangerous hallucinations.

Fallback Behaviors of LLMs Under Uncertainty

The paper "From Loops to Oops: Fallback Behaviors of LLMs Under Uncertainty" by Ivgi et al. provides a comprehensive examination of the fallback behaviors that LLMs exhibit when faced with uncertainty. The authors propose that such behaviors—namely sequence repetitions, degenerate text, and hallucinations—are interconnected and arise when models are uncertain about their outputs. They present a rigorous analysis using models from different families, including Pythia, Llama, and OLMo, which vary in terms of pretraining tokens, parameter count, and instruction-following training.

Categorization of Fallback Behaviors

The authors categorize fallback behaviors into three primary types:

• Sequence Repetitions: Models repeat previously generated sequences within the context.
• Degenerate Text: Models generate repetitive textual patterns or rephrase previously generated content.
• Hallucinations: Models produce coherent but factually incorrect or unfounded content.

Key Findings

1. Scaling and Fallback Behaviors

The analysis demonstrates that as the number of parameters and the amount of training data increases, models shift from simpler fallback behaviors such as repetitions to more complex ones like hallucinations. For instance, in the Pythia model family, larger models produce fewer repeated sequences and more hallucinations (Fig. \ref{fig:main-params}). This trend is consistent across different datasets, including TriviaFacts and Qampari.

2. The Effect of Pretraining

Investigating Pythia-6.9B and Pythia-12B checkpoints reveals that models shift their fallback behaviors as they see more training tokens. Early in the training process, models predominantly repeat sequences, but as training progresses, they start producing more correct answers and hallucinations (Fig. \ref{fig:main-tokens}).

3. Instruction-Tuning

Instruction-tuned models, such as those in the Llama and OLMo families, exhibit a higher propensity for hallucinations relative to their untuned counterparts. Instruction tuning aligns models more closely with human expectations but also amplifies hallucination rates (Fig. \ref{fig:full-type-greedy-hq}).

Practical and Theoretical Implications

Practical Implications

The study's findings suggest that developers of LLMs need to be mindful of the design trade-offs when scaling models or incorporating instruction tuning. Although scaling and instruction tuning improve overall performance and alignment with user expectations, they also make models more prone to generating undetected hallucinations, potentially leading to misuse or misinformation.

The paper also stresses the limitations of common decoding techniques like random sampling. While such methods can mitigate simpler repetitive behaviors (Fig. \ref{fig:temperature}), they tend to increase more dangerous hallucinations.

Theoretical Implications

From a theoretical perspective, understanding that fallback behaviors are inherent to the generative process of LLMs under uncertainty sheds light on the internal mechanisms of these models. This insight can guide future research into developing more robust models that can better handle uncertainty without resorting to undesirable behaviors.

Future Directions

The paper opens several pathways for future research, including:

• Exploring more sophisticated decoding algorithms that can balance between avoiding repetitive sequences and minimizing hallucinations.
• Investigating mechanisms to improve the internal calibration of LLMs to align their confidence with their factual correctness, potentially reducing hallucinations.
• Extending the analysis to other generative tasks, such as code generation, where the stakes for incorrect outputs can be exceptionally high.

Conclusion

Overall, Ivgi et al. provide a significant contribution to the understanding of fallback behaviors in LLMs. The detailed empirical analysis across multiple model families and training paradigms unveils consistent patterns in how models handle uncertainty. This work underscores the need for continued research into more effective methods of controlling LLM outputs to minimize undesirable behaviors while maximizing their utility in practical applications.