Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models

Abstract: The capabilities and limitations of LLMs have been sketched out in great detail in recent years, providing an intriguing yet conflicting picture. On the one hand, LLMs demonstrate a general ability to solve problems. On the other hand, they show surprising reasoning gaps when compared to humans, casting doubt on the robustness of their generalisation strategies. The sheer volume of data used in the design of LLMs has precluded us from applying the method traditionally used to measure generalisation: train-test set separation. To overcome this, we study what kind of generalisation strategies LLMs employ when performing reasoning tasks by investigating the pretraining data they rely on. For two models of different sizes (7B and 35B) and 2.5B of their pretraining tokens, we identify what documents influence the model outputs for three simple mathematical reasoning tasks and contrast this to the data that are influential for answering factual questions. We find that, while the models rely on mostly distinct sets of data for each factual question, a document often has a similar influence across different reasoning questions within the same task, indicating the presence of procedural knowledge. We further find that the answers to factual questions often show up in the most influential data. However, for reasoning questions the answers usually do not show up as highly influential, nor do the answers to the intermediate reasoning steps. When we characterise the top ranked documents for the reasoning questions qualitatively, we confirm that the influential documents often contain procedural knowledge, like demonstrating how to obtain a solution using formulae or code. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy that synthesises procedural knowledge from documents doing a similar form of reasoning.

• The paper reveals that LLMs rely on procedural knowledge in pretraining to drive reasoning across model scales (7B and 35B parameters).
• It shows that reasoning tasks use a broad, integrated synthesis of procedural cues instead of specific document-based answer retrieval.
• The study highlights the pivotal role of code and mathematical data in embedding procedural representations for improved reasoning performance.

Procedural Knowledge in Pretraining Drives Reasoning in LLMs

The research paper, "Procedural Knowledge in Pretraining Drives Reasoning in LLMs," investigates the foundational mechanisms underpinning reasoning capabilities in LLMs. This paper provides a comprehensive examination of the dependency of LLMs on procedural knowledge in pretraining data to perform reasoning tasks as opposed to factual information retrieval, with analyses conducted on two model sizes, 7B and 35B parameters, trained on the same corpus.

Key Findings

The study explores the differential data reliance of LLMs when engaging in reasoning versus factual query answering. Key observations can be summarized as follows:

1. Procedural Knowledge Consistency: The authors highlight that procedural knowledge, particularly for mathematical reasoning tasks, surfaces more consistently across related queries. This is evidenced by a significant correlation between document influence scores for queries involving similar reasoning tasks. This indicates that certain pretraining documents offer generalizable patterns applicable across various instances of a task type, such as arithmetic operations or calculating slopes, which are crucial for LLMs when reasoning.
2. Magnitude and Volatility of Influence: The analysis shows that reasoning largely involves a broader and less volatile spectrum of influences per unit of information generated compared to factual questions. For factual queries, highly specific documents are often influential, suggesting a retrieval-based approach to answering. On the contrary, reasoning queries demonstrate dispersed influence, implying that models utilize a more integrated synthesis of procedural knowledge.
3. Absence of Direct Answer Retrieval in Reasoning: When examining the highly influential documents in the context of reasoning queries, the direct presence of answers— as often observed in factual queries— is significantly limited. Instead, documents exert influence by embedding procedurally relevant information, such as snippets of code or equations that encapsulate the essence of the reasoning task without directly stating the answer.
4. Role of Code and Mathematical Data: The absence of direct answer retrieval in reasoning tasks points toward a reliance on documents featuring procedural representations, such as mathematical operations laid out in code or structured mathematical data. This suggests that code data is critically influential, and models integrate such knowledge to emulate reasoning as opposed to direct answer extraction.

Implications and Future Directions

This study reveals foundational insights into how LLMs generalize learned procedures to tackle reasoning tasks, pointing to a model behavior much closer to synthesis based on procedural learning rather than mere retrieval. Such findings implicate that enhancing the breadth and depth of procedural knowledge in pretraining datasets could bolster model capabilities in reasoning.

Practically, the findings inform pretraining data selection strategies, emphasizing quality and diversity in procedural datasets over mere quantity, potentially optimizing model efficiency and reasoning fidelity. Moreover, the overrepresentation of code in influential reasoning documents hints at an avenue for specializing LLMs via enhanced integration of coding and mathematical data to underpin complex reasoning capabilities.

Theoretically, these results underscore a departure from typical paradigms of answer retrieval towards an integrated, procedural synthesis approach. This provides a promising outlook on the training methodologies for LLMs, advocating for finely-tuned curricula that facilitate procedural knowledge embedding— potentially unlocking enhanced reasoning output across diverse domains.

Future research could explore this procedural synthesis beyond mathematical reasoning into more abstract reasoning tasks, potentially expanding the application and robustness of LLM reasoning. Additionally, understanding the interplay between procedural knowledge influence and model scaling can offer further insights into optimizing model architectures for nuanced reasoning tasks across varying scales.

In conclusion, the findings advocate a strategic rethinking of pretraining paradigms, with an emphasis on procedural knowledge as pivotal for fostering enhanced reasoning capabilities within LLMs. This research sets a robust foundation for future explorations into the depths of procedural generalization and its computational manifestations in AI systems.