Self-Distillation Zero: Self-Revision Turns Binary Rewards into Dense Supervision

Abstract: Current post-training methods in verifiable settings fall into two categories. Reinforcement learning (RLVR) relies on binary rewards, which are broadly applicable and powerful, but provide only sparse supervision during training. Distillation provides dense token-level supervision, typically obtained from an external teacher or using high-quality demonstrations. Collecting such supervision can be costly or unavailable. We propose Self-Distillation Zero (SD-Zero), a method that is substantially more training sample-efficient than RL and does not require an external teacher or high-quality demonstrations. SD-Zero trains a single model to play two roles: a Generator, which produces an initial response, and a Reviser, which conditions on that response and its binary reward to produce an improved response. We then perform on-policy self-distillation to distill the reviser into the generator, using the reviser's token distributions conditioned on the generator's response and its reward as supervision. In effect, SD-Zero trains the model to transform binary rewards into dense token-level self-supervision. On math and code reasoning benchmarks with Qwen3-4B-Instruct and Olmo-3-7B-Instruct, SD-Zero improves performance by at least 10% over the base models and outperforms strong baselines, including Rejection Fine-Tuning (RFT), GRPO, and Self-Distillation Fine-Tuning (SDFT), under the same question set and training sample budget. Extensive ablation studies show two novel characteristics of our proposed algorithm: (a) token-level self-localization, where the reviser can identify the key tokens that need to be revised in the generator's response based on reward, and (b) iterative self-evolution, where the improving ability to revise answers can be distilled back into generation performance with regular teacher synchronization.

• The paper introduces SD-Zero, which transforms binary verification rewards into dense, token-level supervision without external demonstrations.
• The methodology employs a two-phase pipeline—self-revision training followed by self-distillation—to achieve up to 10% performance gains on math and code benchmarks.
• By internalizing self-revision, SD-Zero reduces token consumption and enables LLMs to self-improve, offering a scalable approach for post-training reasoning tasks.

SD-Zero: Transforming Binary Rewards into Dense Self-Supervision for Reasoning LLMs

Motivation and Background

Post-training of LLMs for verifiable tasks (math, code) faces a supervision bottleneck. RL variants exploit binary rewards, but such sparse feedback limits sample efficiency and does not highlight which reasoning steps are incorrect. Dense token-level supervision, as enabled by distillation techniques, typically depends on external teacher models or high-quality demonstrations—resources that may be unavailable when advancing frontier models. SD-Zero introduces a paradigm to leverage only scalar reward signals, with neither external teacher nor curated demonstrations, by training a model to play both generator and reviser roles.

Figure 1: SD-Zero employs an on-policy self-distillation pipeline where revision of incorrect generator responses becomes the teacher’s supervision.

Methodology

SD-Zero’s two-phase pipeline facilitates self-revision and subsequent self-distillation:

Phase 1: Self-Revision Training (SRT)

• The model samples responses as generator, receives a binary correctness reward, and is prompted to revise if incorrect or rephrase if correct.
• Only successful revisions (yielding correct final answers) are retained, forming a dataset that includes both initial (potentially incorrect) responses and subsequent self-revisions.
• Training objective includes both the revision task (generator + initial attempt + reward prompt → revised response) and generation task (generator → correct response from scratch).
• This phase elicits explicit self-revision behaviors and bootstraps reviser capacity.

Phase 2: Self-Distillation (SD)

• The trained SRT model is used as both student (generator) and teacher (reviser).
• The student generates responses; the teacher conditions on the response and binary reward, producing token-level distributions for revision.
• Student is trained to match reviser distributions via in-sequence KL minimization, distilling revision-aware behaviors into generation.

  Figure 2: SD-Zero overview—Phase 1 collects outcome-conditioned revision traces; Phase 2 distills revision into efficient generation.

Empirical Results

SD-Zero was evaluated on Qwen3-4B-Instruct and Olmo-3-7B-Instruct across math and code benchmarks, using strictly matched data/sample budgets for all baselines. The following benchmarks were included: AIME24, AIME25, HMMT25, AMOBench, OpenR1-Math, MATH, Codeforces, and LiveCodeBench.

Key numerical results:

• SRT alone (Phase 1) surpasses SFT and RFT—even with less data—by up to $9.2\%$ on Olmo and $7.8\%$ on Qwen.
• Full SD-Zero (Phase 2) adds further gains ($+2.7\%$ for Qwen, $+1.2\%$ for Olmo), achieving at least $10\%$ improvement over baseline, and consistently outperforming GRPO and SDFT by at least $4.8\%$.
• SD-Zero halves inference-time token consumption relative to SRT and baselines.

  Figure 3: Reviser transforms binary outcome reward into dense, localized token-level reward—incorrect responses concentrate KL on error tokens, correct responses yield flatter distributions.

Behavioral and Analytical Insights

SD-Zero’s core innovation is turning a single binary reward into dense, token-level supervision via model-based self-revision. Analyses reveal:

• Token-Level Self-Localization: The reviser provides sharply localized feedback, concentrating KL divergence signal on a small subset of tokens relevant to error correction. This enables efficient credit assignment not available in conventional RL schemes.
• Iterative Self-Evolution: The model’s ability to revise improves across self-distillation rounds; synchronizing the teacher with updated student parameters sustains performance gains, providing evidence of iterative self-improvement.
• Internalization of Self-Revision: During SRT, explicit self-revision language and response lengths increase. Self-distillation phase suppresses overt revision keywords and reduces length, indicating transition from explicit to internalized reasoning behaviors.

  Figure 4: Left—example traces show transition from failed, verbose answers to internalized, concise guidance; Right—metrics track rise and fall of revision behaviors and response lengths across training phases.

Comparative Ablations

Critical ablations validate that:

• Both revision and generation terms in SRT are required; each alone fails to elicit or transfer self-revision behaviors.
• Applying self-distillation without prior SRT produces negligible revision capability—explicit self-revision must first be bootstrapped before distillation.
• Optimal training budgets favor allocating more data to self-distillation once SRT has unlocked reviser behaviors.
• SDFT and GRPO, under final-answer-only reward, do not recover SD-Zero’s gains; SD-Zero uniquely leverages incorrect trajectory conditioning.

Practical and Theoretical Implications

Practically, SD-Zero enables efficient post-training of LLMs for complex reasoning tasks without dependency on privileged or external knowledge. Sample cost and compute requirements are reduced, with substantial accuracy and token efficiency gains. Theoretically, SD-Zero demonstrates that LLMs can be trained to transform sparse reward signals into dense, targeted supervision, internalizing self-guidance and error correction behaviors. The explicit conditioning on incorrect attempts, and iterative teacher-student synchronization, lead to sustainable self-evolution—potentially applicable to broad model families and domains.

The token-level credit assignment parallels process reward models, but SD-Zero achieves it using only binary rewards, eliminating the need for external step-level labels or auxiliary models. This may offer a viable path for scaling dense reasoning supervision in settings where process annotation is infeasible.

Open directions include:

• Extension to "thinking models" that generate lengthy exploratory trajectories; distinguishing genuine errors from exploratory steps remains unsolved.
• Application to unverifiable domains, where binary rewards are unavailable—potential augmentation via meta-cognitive signals, e.g., consistency or self-correction.
• Integrating SD-Zero with RL or search-based process reward models to further improve credit assignment and exploration.

Conclusion

SD-Zero establishes a self-distillation pipeline that turns binary verification rewards into dense token-level supervision. By conditioning the teacher on initial, possibly incorrect student attempts, SD-Zero facilitates efficient self-improvement without external demonstrations, achieving $10\%+$ performance gains and substantial token savings relative to baselines. Extensive ablations confirm that the flow from explicit self-revision to internalized reasoning relies on complementary loss terms, teacher synchronization, and targeted credit assignment. The method is readily extensible for diverse verifiable reasoning tasks, and portends further advances in self-improving LLM architectures.