In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly

Abstract: In-context learning (ICL) enables transformers to adapt to new tasks through contextual examples without parameter updates. While existing research has typically studied ICL in fixed-complexity environments, practical LLMs encounter tasks spanning diverse complexity levels. This paper investigates how transformers navigate hierarchical task structures where higher-complexity categories can perfectly represent any pattern generated by simpler ones. We design well-controlled testbeds based on Markov chains and linear regression that reveal transformers not only identify the appropriate complexity level for each task but also accurately infer the corresponding parameters--even when the in-context examples are compatible with multiple complexity hypotheses. Notably, when presented with data generated by simpler processes, transformers consistently favor the least complex sufficient explanation. We theoretically explain this behavior through a Bayesian framework, demonstrating that transformers effectively implement an in-context Bayesian Occam's razor by balancing model fit against complexity penalties. We further ablate on the roles of model size, training mixture distribution, inference context length, and architecture. Finally, we validate this Occam's razor-like inductive bias on a pretrained GPT-4 model with Boolean-function tasks as case study, suggesting it may be inherent to transformers trained on diverse task distributions.

• The paper reveals that transformers automatically select the simplest sufficient hypothesis in in-context tasks using a Bayesian framework.
• It validates Occam’s razor behavior empirically across hierarchical testbeds including Markov chains, linear regression, and PCFGs.
• The findings imply that training on mixed task complexities enhances transformer adaptability and interpretability in ambiguous settings.

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly

This paper presents a systematic investigation into the inductive biases of transformers in in-context learning (ICL) scenarios where tasks are organized hierarchically by complexity. The central claim is that transformers, when trained on mixtures of tasks with varying complexity, consistently select the simplest hypothesis sufficient to explain the in-context data, rather than defaulting to the most expressive available model. This behavior is theoretically justified via a Bayesian framework, and is empirically validated across synthetic and real-world settings, including Markov chains, linear regression, probabilistic context-free grammars (PCFGs), and large pretrained LLMs such as GPT-4.

Problem Setting and Motivation

ICL enables transformers to adapt to new tasks by conditioning on contextual examples, without parameter updates. Prior work has largely focused on fixed-complexity tasks, but real-world applications present a spectrum of task complexities. The authors address the question: When presented with data compatible with multiple hypothesis classes, do transformers select the simplest sufficient hypothesis, or do they default to the most complex available?

To probe this, the authors construct controlled testbeds where higher-complexity task classes strictly contain lower-complexity ones (e.g., higher-order Markov chains can represent all lower-order chains). This setup introduces inherent ambiguity: for data generated by a simple process, both simple and complex hypotheses can explain the data perfectly.

Experimental Framework

Markov Chains

Transformers are trained on sequences generated by both order-1 (simple) and higher-order (complex) Markov chains. At inference, the model is prompted with sequences from either class. The key metric is the KL divergence between the model's output distribution and the empirical $n$-gram statistics of the context (e.g., bigram for order-1, tetragram for order-3).

Findings:

• The transformer accurately infers the true order of the generating process from the context.
• When prompted with order-1 data, the model's predictions align with bigram statistics, not higher-order statistics, despite the latter being expressive enough to fit the data.
• When prompted with higher-order data, the model switches to the appropriate higher-order statistics.

Linear Regression

A similar hierarchy is constructed for linear regression: the "simple" category consists of regressors in a lower-dimensional subspace, while the "complex" category uses the full feature space. Both categories can perfectly fit data generated by the simple regressor.

Findings:

• When prompted with data from the simple regressor, the transformer aligns its predictions with the lower-dimensional least-squares solution, not the full-dimensional one.
• For complex data, the model uses the full-dimensional solution.

Probabilistic Context-Free Grammars (PCFGs)

Transformers trained on mixtures of simple and complex PCFGs can infer the type and parameters of the generating grammar from the context, again favoring the simplest sufficient explanation.

Pretrained LLMs (GPT-4)

Prompting GPT-4 with Boolean function tasks that admit both simple and complex explanations, the model consistently selects the simple function when the context is ambiguous, and the complex function only when necessary.

Theoretical Analysis

The authors provide a Bayesian justification for the observed behavior. For both Markov chains and linear regression, the Bayes-optimal predictive distribution is a mixture over hypothesis classes, weighted by their posterior probabilities given the context. The marginal likelihood for each class decomposes into a data fit term and a complexity penalty (akin to BIC). When the data is compatible with multiple classes, the complexity penalty ensures that the posterior concentrates on the simplest sufficient class—implementing a form of Bayesian Occam's razor.

Key formula (Markov chains):

$$\log p(X|s) \approx \sum_t \log \hat{p}_X(x_t | x_{t-1}, ..., x_{t-s}) - \frac{V^s(V-1)}{2} \log T$$

The first term is the empirical likelihood; the second is a complexity penalty increasing with model order.

Ablations and Additional Results

• Training on only the complex class: Transformers trained solely on the most complex class do not generalize to lower-complexity statistics at inference, indicating that the Occam's razor bias emerges only when the training distribution includes multiple complexity levels.
• Effect of training mixture proportion: The model reliably learns simple statistics even when simple tasks are a minority in the training mix, but learning complex statistics becomes harder as the mix becomes more imbalanced.
• Model scale: Larger transformers converge faster and more reliably to the correct statistics for both simple and complex tasks.
• Comparison with LSTMs: LSTMs require significantly more capacity to exhibit similar inductive bias, and their Occam's razor behavior is weaker.

Implications

Practical

• Robustness in Real-World ICL: The Occam's razor inductive bias suggests that transformers can robustly adapt to tasks of unknown complexity, favoring parsimonious explanations unless the data demands otherwise. This is beneficial for generalization and sample efficiency in practical deployments.
• Task Mixture Design: For applications requiring flexible adaptation, training on a diverse mixture of task complexities is essential to induce this bias.
• Model Selection and Interpretability: The Bayesian framework provides a principled lens for interpreting transformer predictions in ambiguous settings, with potential applications in model auditing and interpretability.

Theoretical

• Mechanistic Understanding: The results motivate further investigation into the internal mechanisms by which transformers implement complexity selection, including the role of attention heads and architectural components.
• Optimization Dynamics: The emergence of Bayesian selection principles from gradient-based training dynamics remains an open question.
• Extension to Richer Hierarchies: Future work could explore more complex hierarchical structures, such as tasks with multiple axes of complexity or structural constraints beyond dimensionality.

Future Directions

• Mechanistic interpretability: Dissecting the specific circuits and attention patterns responsible for complexity selection.
• Optimization theory: Analyzing how training dynamics give rise to Bayesian Occam's razor in deep networks.
• Broader task hierarchies: Extending the analysis to tasks with non-nested or multi-dimensional complexity relationships.
• Transfer to real-world data: Validating the inductive bias in large-scale, naturalistic settings beyond synthetic testbeds.

Conclusion

This work provides strong empirical and theoretical evidence that transformers trained on mixtures of tasks with hierarchical complexity exhibit a robust Occam's razor inductive bias in-context. This property is not only theoretically appealing but also practically advantageous for generalization and adaptability in diverse real-world scenarios. The findings have significant implications for the design, training, and interpretation of transformer-based models in both research and applied settings.