Self-Distilled Reasoner in LLMs

• Self-distilled reasoners are LLMs that improve reasoning by using self-supervised privileged traces, eliminating the need for external teacher models.
• They employ on-policy distillation, self-labelling, and iterative refinement techniques to enhance token efficiency and accuracy on complex reasoning tasks.
• Empirical evaluations demonstrate significant gains in sample efficiency, benchmark accuracy, and convergence stability through dense per-token supervision.

A self-distilled reasoner is a LLM or a smaller-scale variant trained or evolved to improve its reasoning ability by leveraging its own outputs or privileged traces, rather than relying on external teacher models. Recent frameworks operationalize self-distillation via various regimes—explicit on-policy divergence minimization, progressive self-labelling, reinforcement learning, or self-synthesis of chain-of-thought (CoT) rationales—combined with privileged contexts or automated filtering. These approaches address key drawbacks of traditional supervised fine-tuning and off-policy distillation, namely distribution mismatch and inefficiency in utilizing available high-quality reasoning traces. Self-distilled reasoners demonstrate enhanced sample efficiency, improved solve rates on complex reasoning tasks, and robustness even with limited teacher access or model scale constraints (Zhao et al., 26 Jan 2026).

1. Motivation and Conceptual Foundations

Self-distillation in LLMs originates from the observation that reasoning performance plateaus with standard supervised fine-tuning (SFT) and teacher-student knowledge distillation due to finite teacher compute, labor-intensive ground-truth annotation, and mode collapse on student rollouts unseen during training. SFT suffers from off-policy distribution mismatch: the student model is exposed only to oracle trajectories, causing compounding errors at deployment when it generates or samples alternative prefixes. On-policy distillation mitigates this by aligning the student’s actual generation policy with a teacher or self-supervised target, typically through dense, per-token feedback (Zhao et al., 26 Jan 2026).

Self-distilled reasoning differs from earlier methods in that the model itself—conditioned on privileged information, more context, or specific meta-reasoning prompts—acts as its own teacher. Thus, self-distillation frameworks train models by providing supervision based on their own ability to rationalize gold or synthesized reasoning traces, exploiting properties such as model calibration, trajectory diversity, and internalization of latent reasoning capacities (Zhang et al., 18 Feb 2025, Wang et al., 20 May 2025).

2. On-Policy Self-Distillation Objective

The formalism for on-policy self-distillation centers on defining two conditional policies from the same parameterization: a student policy $\pi_S$ (inference policy) conditioned only on the problem $x$, and a teacher policy $\pi_T$ conditioned on both $x$ and the privileged trace $y^*$. For a reasoning dataset $\mathcal{S} = \{(x, y^*)\}$ and student rollout $\hat y \sim \pi_S(\cdot \mid x)$, the self-distillation loss is the expected full-vocabulary per-token divergence: $$\mathcal{L}_{\mathrm{OPSD}}(\theta) = \mathbb{E}_{(x,y^*) \sim \mathcal{S}}\,\mathbb{E}_{\hat y \sim \pi_S(\cdot|x)}\left[ \frac{1}{|\hat y|} \sum_{n=1}^{|\hat y|} D\bigl(\pi_T(\cdot|x,y^*,\hat y_{<n})||\pi_S(\cdot|x,\hat y_{<n})\bigr) \right]$$ where $D$ is typically Jensen–Shannon or KL divergence. Supervision is provided at every token along the student’s own trajectory, with the teacher’s distribution held fixed to stabilize learning. This framework avoids reliance on external teacher LLMs or selection of generated pseudo-labels (Zhao et al., 26 Jan 2026).

This mechanism is distinct from conventional self-distillation in image models (e.g., Hinton et al.) and from reinforcement learning with reward-based updates (e.g., RLVR/GRPO), as it provides dense, privileged information at the token level, preserving the semantics of high-quality ground-truth traces.

3. Algorithmic Frameworks and Variants

Self-distilled reasoning frameworks have diversified along technical and application axes:

• On-Policy Self-Distillation (OPSD): A single LLM generates rollouts on its own policy and receives per-token divergence supervision from a version of itself conditioned on gold reasoning traces. Empirically, this leads to higher token efficiency (4–8x fewer generated tokens than RL methods like GRPO), as well as accuracy gains on AIME24/25, HMMT25, and AMO-Bench (Zhao et al., 26 Jan 2026).
• Self-Enhanced Reasoning Training (SERT): For small models, SERT surfaces latent reasoning paths by sampling multiple alternatives under zero-shot prompting, filtering based on sequence quality (pattern, length, repetition, perplexity), and fine-tuning on these pseudo-labels before teacher-guided distillation. This yields monotonic improvements in reasoning tasks (e.g., StrategyQA, CommonsenseQA), and the ablation confirms the effect of each filter (Zhang et al., 18 Feb 2025).
• Self-Reasoning LLMs (SRLM): SRLM bootstraps more elaborate CoT traces by iteratively synthesizing and selecting longer or more effective rationales, guided by a few demonstration “catalyst” examples crafted with meta-reasoning instructions. At each iteration, the model selects the best rationale among new samples, growing its internal repository of advanced, diverse reasoning patterns. This approach yields average accuracy boosts of $2.5+$ points across multiple benchmarks and scales with increasing sampling factor (Wang et al., 20 May 2025).
• Reinforcement Learning and Self-Taught Reasoners (STaR/RL-STaR): Here, reasoning is improved by letting the model sample, filter, and fine-tune on its own generated chains where the final answer is correct. Theoretical analysis shows each such round provably increases the alignment of reasoning transitions with ground-truth, given a minimally competent pre-trained model (Chang et al., 2024).
• Native Parallel Reasoner (NPR): NPRs extend self-distillation to the parallel regime, with models learning through progressive curriculum (format discovery, constrained SFT, and native parallel RL) to generate decomposed, parallel-executable reasoning steps. A key innovation is Parallel-Aware Policy Optimization (PAPO), which encourages agentic branching within an execution graph and exploits self-distillation by retaining only trajectories that satisfy both form and correctness constraints (Wu et al., 8 Dec 2025).
• Iterative Test-Time Self-Distillation (DSER): DSER runs parallel long-horizon chains of refinement and verification, treating the self-correction process as a Markov chain. Majority vote across rollouts amplifies small self-improvement probabilities until convergence, even in open-weight models with weak verification. This variant outperforms large teacher models in test-time majority settings (Liu et al., 20 Oct 2025).

4. Empirical Evaluations and Benchmarks

Self-distilled reasoners demonstrate quantitative improvements across competitive programming, mathematics, and general reasoning benchmarks. In OPSD, Qwen3-8B achieves average@16 accuracy of 52.2% on target math datasets, exceeding SFT (50.0%) and off-policy RL methods (GRPO: 51.3%) under equivalent compute. Token efficiency is increased by a factor of 4–8x, reducing training cost per accuracy gain (Zhao et al., 26 Jan 2026). SERT improves the zero-shot and distillation performance of GPT-2 on StrategyQA and CommonsenseQA by up to 3–5 percentage points (Zhang et al., 18 Feb 2025).

SRLM produces substantial incremental accuracy increases by scaling inference-time sampling ($+7.89$ points with $N=64$) and demonstrates the effectiveness of a small number of meta-reasoning “catalyst” examples (Wang et al., 20 May 2025). DSER achieves Cons@64 accuracy of 89.3% (AIME 2024) and is able to solve “previously unsolvable” problems by leveraging deep self-correction (Liu et al., 20 Oct 2025). NPR achieves state-of-the-art performance and 4x–5x inference speedup on parallel-structured math and logic tasks, validating both correctness and efficiency improvements from self-distilled parallel policies (Wu et al., 8 Dec 2025).

5. Theoretical Analysis and Convergence Guarantees

The theoretical foundations for self-distilled reasoning are formalized in RL-STaR, where it is shown that, given a pre-trained LLM whose stepwise reasoning transitions are “better than random” (i.e., each step has bias $\delta_0 > 0$ toward correctness), self-improvement is guaranteed at each STaR round. The per-iteration policy improvement theorem shows $\delta_t > \delta_{t-1}$, and in the infinite data and iteration limit, the process converges to the ground-truth policy ($\delta_t \to 1–1/M$, $P_t \to I_M$). The framework is robust to occasional incorrect intermediate steps, as these are exponentially suppressed over successive iterations (Chang et al., 2024).

For iterative schemes such as DSER, convergence to the correct solution is guaranteed under the condition that each reasoning refinement has a higher probability of improvement than degradation ($p_{IC} > p_{CI}$), with the stationary distribution and mixing rates exactly characterized for the associated Markov chain process (Liu et al., 20 Oct 2025). In SRLM, empirical evidence supports that iterative rationale expansion—anchored by catalyst demonstrations—produces more diverse and meta-reasoned solutions over time (Wang et al., 20 May 2025).

6. Limitations and Open Challenges

Several practical and theoretical limitations are highlighted. OPSD and related frameworks have been validated only up to 8B-parameter models; it remains open whether similar gains are reproducible for 70B+ architectures (Zhao et al., 26 Jan 2026). Teacher signal quality degrades on tasks beyond initial model comprehension or tractable gold reasoning trace availability, motivating research in curriculum and adaptive verification. Filtering-based protocols (as in SERT) are naturally sensitive to chosen thresholds and increase sample complexity, introducing computational burdens on large-scale corpus construction (Zhang et al., 18 Feb 2025). Test-time iterative schemes such as DSER trade budget for model capacity and do not yield improved single-turn inference.

Another open challenge is integrating explicit correctness verification or external reward signals into self-distillation frameworks. Current loss formulations focus on matching distributions or traces but do not reward final answer accuracy or logical coherence directly. Approaches combining binary verification, reward-augmented objectives, or stronger self-verification modules are proposed avenues for further research (Zhao et al., 26 Jan 2026, Wu et al., 8 Dec 2025).

7. Comparative Synthesis of Major Self-Distilled Reasoning Frameworks

Framework	Core Mechanism	Setting	Key Metric/Result
OPSD (Zhao et al., 26 Jan 2026)	On-policy, privileged per-token distillation	Math reasoning	4–8x token efficiency over RL
SERT (Zhang et al., 18 Feb 2025)	Self-labelling latent CoT traces + filter	Small models	+3–5 pts on StrategyQA, CommonsenseQA
SRLM (Wang et al., 20 May 2025)	Iterative self-expansion with few catalysts	General reasoning	+2.5 pts avg, +8.2 pts at N=16 samples
DSER (Liu et al., 20 Oct 2025)	Markovian iterative majority-vote process	Math/Olympiad	89.3% Cons@64, >600B teacher
NPR (Wu et al., 8 Dec 2025)	Parallel curriculum, self-distilled RL	Parallel math	Up to +24.5% accuracy, 4.6x speedup
RL-STaR (Chang et al., 2024)	Theoretical RL self-distillation	General CoT	Guaranteed monotonic $\delta_t$

Each approach exploits model-internal generation, privileged traces, or self-evaluation for iterative reasoning improvement, with empirical and theoretical support for monotonic capability gains and increased robustness.

---

Collectively, these developments establish the self-distilled reasoner framework as a foundational paradigm for upgrading LLM reasoning performance without reliance on external teachers. By engineering algorithms that exploit model self-knowledge (privileged context, latent traces, meta-reasoning demonstrations) in on-policy, RL, and self-supervised regimes, research has achieved gains in efficiency, reasoning diversity, robustness, and accuracy across a spectrum of complex problem domains.