Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition

Abstract: This paper investigates knowledge distillation from a large reasoning model (DeepSeek-R1) to a compact student model (Qwen2.5-7B). Using historical problems from the John O'Bryan Mathematics Competition at Northern Kentucky University (2011-2025), we build a Chain-of-Thought (CoT) training corpus through a dual-agent framework. The dataset is used to fine-tune the student model with Low-Rank Adaptation (LoRA) on Apple Silicon hardware using the MLX framework. The base Qwen2.5-7B model achieves 64.67% accuracy on competition problems, while the DeepSeek-R1 teacher achieves 91.40%. An initial 1,000-iteration training run revealed severe overfitting, with validation loss reaching a minimum at iteration 200 before rising steadily. Based on this finding, we ran five independent training runs each limited to 200 iterations with varied random seeds to assess result stability. Across these five runs, the fine-tuned student model achieves a mean accuracy of 69.43% (std dev 0.17%) on the competition dataset, a 4.76 percentage-point improvement over the base model, and generalizes to 73.1% (std dev 0.18%) on the MATH-500 benchmark. We further study how response length affects answer quality across six reasoning levels (R1-R6): accuracy declines consistently from 69.43% at R1 (mean 220 words) to 41.9% at R6 (mean 31.2 words), with the two-person speed section most sensitive to token reduction. These results demonstrate that CoT distillation improves compact student models and that response length is a critical factor in mathematical reasoning quality.

• The paper proposes a novel CoT distillation method that transfers mathematical reasoning from DeepSeek-R1 to Qwen2.5-7B, achieving a 4.76% accuracy improvement.
• It employs a dual-agent teacher-verifier framework on a curated set of 671 competition problems with early stopping to counteract overfitting.
• The study quantifies how reducing token budgets degrades solution accuracy, offering actionable insights for balancing inference speed and reasoning fidelity.

Knowledge Distillation for Mathematical Reasoning: CoT Transfer from DeepSeek-R1 to Qwen2.5-7B on Competition-Style Problems

Introduction

The paper "Knowledge Distillation from Large Reasoning Models to Compact Student Models: A Case Study on the John O Bryan Mathematics Competition" (2606.31048) presents a systematic investigation into the distillation of mathematical reasoning skills from a teacher LLM (DeepSeek-R1) to a compact student model (Qwen2.5-7B) using a domain-specific dataset. The authors develop a Chain-of-Thought (CoT) corpus from 15 years of John O'Bryan Mathematics Competition problems, employing a dual-agent teacher-verifier structure. The goal is to enhance the student model’s mathematical reasoning ability, evaluate generalization across problem types, and measure the effects of response length on solution accuracy.

Methodology and Experimental Design

The study begins with careful dataset curation: 671 competition problems from the years 2011–2025 are digitized and annotated, with DeepSeek-R1 providing both CoT solutions and independent answer verification. Only problems for which the teacher produces a verified-correct chain are retained for distillation training. The dataset is split chronologically to eliminate potential data leakage.

For distillation, the teacher operates in a dual-agent mode—one agent generating CoT traces and the other serving as a ground-truth-based verifier. The student (quantized Qwen2.5-7B-Instruct) undergoes LoRA-based fine-tuning via the MLX framework. A diagnostic training run demonstrates rapid overfitting, so an early stopping protocol is established: five runs of 200 iterations each with distinct seeds ensure stabilty and reproducibility. The evaluation regime involves both in-domain (competition) and out-domain (MATH-500) problem sets, with the teacher model verifying all student solutions.

Figure 1: Average problem difficulty by year (DeepSeek-R1 ratings, scale 1--10). Darker bars are at or above the mean (5.19).

Additionally, a multi-level token budget experiment is designed to systematically study how compressing response length (in words/tokens) affects accuracy across six inference levels (R1–R6), corresponding to decreasing allowed output lengths.

Training Dynamics and Stability

A prominent finding is the overfitting behavior during fine-tuning on the relatively small, highly structured mathematics corpus. Training loss falls precipitously, while validation loss minimizes early (iteration 200) before increasing due to memorization rather than generalization.

Figure 2: Training vs.\ validation loss over 1,000 iterations. Validation loss reaches its minimum (0.374) at iteration~200 (dotted line), indicating overfitting on a small corpus beyond this point.

Early stopping is demonstrated to be essential for model generalizability. Across five seeds, the accuracy improvement remains consistent (69.43% ± 0.17%), confirming robust training dynamics and the absence of initialization artifacts.

Results: Distillation Effectiveness and Generalization

Distillation from DeepSeek-R1 (teacher) to Qwen2.5-7B yields a significant improvement in student accuracy, with the following numerical outcomes:

• Base student (Qwen2.5-7B): 64.67% accuracy on competition problems
• After CoT distillation (200-iteration runs): 69.43% ± 0.17%
• DeepSeek-R1 teacher: 91.40%
• Out-of-domain (MATH-500) generalization: student improves from 70.1% (base) to 73.1% ± 0.18% post-distillation

Section-level analysis demonstrates the largest relative improvement in competition areas congruent with the training corpus (Algebra and Number Theory). Subject transferability remains limited in underrepresented domains such as intermediate algebra and geometry.

The authors observe that approximately 40% of failures are due to answer formatting rather than fundamental reasoning errors, suggesting that accuracy could be further enhanced via improved output parsing or light post-processing.

Token Budget and Reasoning Quality

A key experimental contribution is the meticulous study of how varying response length constraints degrade solution accuracy. Results show a systematic drop as the allowed token budget is reduced:

• R1 (unrestricted response): 220 words; 69.43%
• R2 (400 tokens): 119.8 words; 61.9%
• R6 (100 tokens): 31.2 words; 41.9%

The most dramatic declines occur below approximately 50–100 words, establishing a practical lower bound for effective multi-step mathematical reasoning within compact response budgets. The two-person speed section is markedly more sensitive to length constraints, showing a 34.5 point drop between R2 and R6, reflecting the necessity of explicit multi-step calculations in such problems.

Figure 3: Accuracy by year and reasoning level (correct_flag == 1 questions only).

Yearly and section-level analysis further reveal that easier problem years allow for more accurate compressed reasoning, while harder years’ accuracy collapses at lower token budgets.

Theoretical and Practical Implications

The experimental pipeline validates the transferability of CoT-based distillation on a domain-specific, expert-annotated corpus with low contamination risk. Results empirically confirm that LoRA-based fine-tuning, when combined with careful early stopping, induces genuine reasoning improvements in compact models, and that the effect is robust across training runs.

The theoretical implications include support for the hypothesis that reasoning ability in LLMs partly resides in step-level CoT representations, and that compact models can reliably absorb such reasoning traces from a teacher even with limited trainable capacity (<0.2% of parameters updated). Further, the sharp impact of token budget constraints quantifies a boundary for compressed mathematical reasoning—highlighting a fundamental trade-off between inference speed/response length and solution fidelity for compact LLMs.

Practically, these findings inform model deployment in educational software and mathematics assistants. Transfer of teacher reasoning can produce smaller, more efficient students suitable for local hardware, but tight response length constraints will substantially reduce solution accuracy—especially for multi-step tasks—unless models are explicitly trained on compressed CoT traces or output post-processing is employed. The identification that 40% of answer failures are non-mathematical points towards tractable engineering solutions to boost measured accuracy.

Future Directions

The authors indicate that future studies should:

• Increase corpus diversity and volume to broaden subject-matter generalization
• Employ stratified sampling across competition sections to balance underrepresented topics
• Experiment with explicit distillation of compressed CoT traces, to maintain accuracy under low token budget scenarios
• Investigate parser-based post-processing and output normalization for non-formatting errors

A further research vector is the development of architecture-agnostic or cross-model LoRA adapters, as current adapters are shown to be incompatible across Qwen parameterizations.

Conclusion

This paper rigorously demonstrates that CoT-based knowledge distillation from a high-performing teacher LLM to a compact student model is both feasible and effective for mathematical reasoning tasks on real-world competition-style data. Achieving a 4.76% absolute accuracy improvement with stable reproducibility, the study establishes both an empirical methodology for compact model refinement and important operational thresholds for solution length in mathematical tasks. The combination of quantitative ablation, error-type classification, and token budget sensitivity analysis offers actionable insights for both theoretical modeling of reasoning transfer and practical system design in mathematical AI applications.