DemoPSD: Disagreement-Modulated Policy Self-Distillation

Abstract: On-policy self-distillation (OPSD) has emerged as a practical method for training LLMs to reason, where a single model acts as both the teacher and the student with different levels of information access. However, recent studies have found that the teacher's dense token-level supervision, conditioned on privileged information, can lead to overfitting to in-domain patterns, suppress exploration, and hurt cross-domain generalization, while also introducing a more fundamental issue: privileged information leakage, where the student encodes answer-dependent shortcuts that are unavailable at test time. We introduce DemoPSD, a novel framework that resolves such problems through the idea of selective adoption of teacher guidance. Instead of fitting the full teacher distribution, DemoPSD steers the student toward a reverse-KL barycenter target, a weighted geometric combination of the teacher and student distributions, that naturally balances learning from the teacher with preserving the student's own reasoning capacity. We measure the difference between their distributions and use such a discrepancy to adaptively control the blending at each token position. We provably show that DemoPSD achieves (1) leakage attenuation, i.e., effective mitigation of privileged information leakage; and (2) exploration preservation, i.e., preservation of exploration capacity under dense token-level distillation. Extensive experiments on SciKnowEval across four scientific fields show that DemoPSD outperforms both GRPO and SDPO while maintaining higher training entropy and robustly generalizing to out-of-distribution GPQA benchmarks.

• The paper introduces DemoPSD, a method that adaptively interpolates between teacher and student distributions to prevent privileged information leakage.
• It leverages a reverse-KL barycenter target to preserve training entropy, leading to up to 4.2% improvement in in-domain accuracy and stable OOD performance.
• The approach provides formal guarantees for leakage attenuation and exploration preservation, offering robust improvements across diverse scientific domains.

DemoPSD: Disagreement-Modulated Policy Self-Distillation

Introduction

DemoPSD introduces a new approach to on-policy self-distillation (OPSD) for training LLMs in scientific reasoning, focusing on mitigating the critical failure mode of privileged information leakage. In standard OPSD, a privileged teacher (conditioned on information unavailable at inference) provides dense token-level supervision to a student, leading to efficient training but also to overfitting to signals that do not generalize and to suppression of the student’s exploration capacity. This work characterizes the origins of this leakage and formally analyzes its impact, before proposing DemoPSD—a method that adaptively interpolates between teacher and student distributions based on their measured local disagreement. This design is motivated by the need to balance leveraging guidance from privileged information and preserving the student’s own general reasoning strategies.

Background and Problem Formulation

In RLVR settings, LLMs are post-trained on a dataset of questions and verifiable answers. Classical RLVR methods such as GRPO provide only trajectory-level signals and suffer from poor token-level credit assignment. OPD addresses this with dense supervision, especially when the privileged teacher can provide rich signals at every token. However, in OPSD, where the same model provides both the teacher (privileged) and student (unprivileged) distributions, the mismatch in available conditioning information (accessible only to the teacher during training) introduces an irreducible mutual information gap. This gap provably causes privileged information leakage: students learn to encode spurious, non-generalizing cues correlated with the privileged information, rapidly improving initially but then degrading both in-domain and out-of-distribution (OOD) accuracy over the course of training (2607.02502).

DemoPSD Approach

DemoPSD’s key innovation is the selective adoption of teacher guidance, modulated by explicit measurement of teacher–student distributional disagreement at each token. The student interpolates between the teacher’s and its own distributions using a reverse-KL barycenter target, where the interpolation coefficient $\alpha_t$ is a monotonic function of the Jensen-Shannon divergence $d_t$ between teacher and student predictions at position $t$:

$$\pi_\text{target}^{\alpha_t}(v) \propto \big(\pi_T^t(v)\big)^{1-\alpha_t} \cdot \big(\pi_S^t(v)\big)^{\alpha_t}$$

Low disagreement leads the student to copy the teacher’s policy, while high disagreement (suggesting heavy teacher dependence on privileged information) graduates the target toward the student, thus attenuating privileged guidance at risky positions.

Figure 2: Distribution of per-token JSD disagreement $d_t$; most tokens have near-zero disagreement, highlighting the token-level sparsity in significant privileged information leakage.

This strategy contrasts with prior works (e.g., entropy gating, consensus routing, or sample-based selection) by directly defining a disagreement-aware distillation target, with formal guarantees for both leakage attenuation and exploration preservation.

Theoretical Analysis

Theoretically, DemoPSD satisfies two primary desiderata:

1. Leakage Attenuation: The expected squared magnitude of the privileged deviation in log-probability space is attenuated by $(1-\alpha_t)^2$ at each token, with larger attenuation at positions of higher teacher–student divergence.
2. Exploration Preservation: The entropy of the DemoPSD target is provably between that of the student and the teacher, and increases as $\alpha_t$ grows. This preserves wider policy exploration at training time, counteracting policy entropy collapse under dense teacher supervision.

Figure 3: Policy entropy over training steps shows DemoPSD maintains significantly higher entropy than SDPO, providing proof of improved exploration.

Experiments

Experiments are conducted on the SciKnowEval benchmark across biology, chemistry, material science, and physics, evaluating three dimensions: in-domain accuracy, training entropy, and OOD generalization (GPQA Extended).

In-Domain Results

DemoPSD leads to strong gains across all metrics (mean@16, maj@16, best@16), with up to 4.2% improvement over SDPO and GRPO. The largest improvements are observed in best@16, indicative of the richer solution diversity resulting from entropy preservation.

Figure 4: Mean@16 accuracy on SciKnowEval; DemoPSD maintains a consistent and increasing lead over SDPO across all scientific domains.

Out-of-Distribution Generalization

Under OOD evaluation on GPQA Extended, DemoPSD avoids the performance collapse characteristic of SDPO, particularly in domains where privileged information leakage is most pronounced. SDPO’s accuracy sharpens early and then degrades, whereas DemoPSD’s OOD accuracy remains stable or improves during continued training.

Figure 1: OOD generalization on GPQA Extended; DemoPSD maintains and improves OOD generalization while SDPO degrades significantly with prolonged training.

Training Dynamics and Disagreement Analysis

DemoPSD preserves 33–98% higher entropy than SDPO at convergence. Disagreement analysis further reveals that high student–teacher disagreement is sparse and localized (only 2–5% of tokens exceed $d_t > 0.25$); thus, DemoPSD’s selective attenuation applies only where privileged leakage is most acute, and dense supervision is preserved everywhere else.

Hyperparameter Sensitivity

DemoPSD’s effectiveness is robust to the choice of the gating sharpness $\beta$, provided $\beta$ is set within a reasonable domain-dependent range. Domains manifesting low average disagreement benefit from higher $d_t$0 (amplifying attenuation), while high-disagreement domains require lower values to maintain stable learning.

Implications and Future Directions

DemoPSD establishes a new framework for OPSD in LLMs where fine-grained, adaptive control over teacher signal adoption at the token level yields both practical accuracy and theoretical generalization improvements. This formulation suggests that broader applications of such agreement-modulated objectives may be fruitful in other settings where privileged information, exposure bias, or overfitting to non-generalizing signals are major concerns. The core principle—guiding dense policy distillation using explicit measurement of local distributional mismatch—offers a direct path to more robust and reliable transfer, especially in settings with high class-imbalance or distribution shift.

Practical deployment of DemoPSD may drive the design of future RLHF, OPD, and curriculum learning protocols for LLMs characterized by a need to balance dense reward shaping with robust generalization. Extending the method to multi-teacher and hierarchical contexts, as well as dynamic adjustment of disagreement sensitivity, are promising avenues for further exploration.

Conclusion

DemoPSD introduces a disagreement-modulated, reverse-KL barycenter target for on-policy self-distillation, enabling robust prevention of privileged information leakage without sacrificing the benefits of dense supervision. It demonstrably preserves exploration entropy, yields strong numerical gains in both in-domain and OOD accuracy, and provides theoretical guarantees unmatched by existing OPSD approaches. The work provides a rigorous and extensible foundation for future research in robust, self-distilled policy optimization across LLMs and related domains.