Privileged Information Distillation for Language Models

Abstract: Training-time privileged information (PI) can enable LLMs to succeed on tasks they would otherwise fail, making it a powerful tool for reinforcement learning in hard, long-horizon settings. However, transferring capabilities learned with PI to policies that must act without it at inference time remains a fundamental challenge. We study this problem in the context of distilling frontier models for multi-turn agentic environments, where closed-source systems typically hide their internal reasoning and expose only action trajectories. This breaks standard distillation pipelines, since successful behavior is observable but the reasoning process is not. For this, we introduce π-Distill, a joint teacher-student objective that trains a PI-conditioned teacher and an unconditioned student simultaneously using the same model. Additionally, we also introduce On-Policy Self-Distillation (OPSD), an alternative approach that trains using Reinforcement Learning (RL) with a reverse KL-penalty between the student and the PI-conditioned teacher. We show that both of these algorithms effectively distill frontier agents using action-only PI. Specifically we find that π-Distill and in some cases OPSD, outperform industry standard practices (Supervised finetuning followed by RL) that assume access to full Chain-of-Thought supervision across multiple agentic benchmarks, models, and forms of PI. We complement our results with extensive analysis that characterizes the factors enabling effective learning with PI, focusing primarily on π-Distill and characterizing when OPSD is competitive.

• The paper introduces π-Distill, a joint teacher-student framework that transfers privileged training signals to improve language model performance without explicit chain-of-thought supervision.
• It combines π-Distill and On-Policy Self-Distillation (OPSD) to stabilize learning through reverse KL regularization and dense, per-token feedback.
• Empirical results show up to 11.8% improvement in agentic tasks across various benchmarks, validating the approach for out-of-domain generalization and efficient knowledge transfer.

Privileged Information Distillation for LLMs: Summary and Implications

Introduction and Motivation

The challenge of transferring capabilities learned using training-time privileged information (PI) to LMs that must operate without this information at inference is a central obstacle for advancing RL-finetuned LMs in complex, multi-turn, agentic environments. Current industry practice often involves distilling closed-source, frontier models by performing supervised fine-tuning (SFT) and subsequent RL on observable action traces. However, leading closed-source systems increasingly occlude their Chain-of-Thought (CoT) reasoning, exposing only final action outputs—fundamentally impairing the effectiveness of standard distillation pipelines that rely on explicit reasoning supervision.

This work introduces $\pi$-Distill, a joint teacher-student learning framework, and On-Policy Self-Distillation (OPSD), as robust methods for leveraging PI (such as action traces or synthesized hints), enabling the student to generalize to settings where PI is unavailable at test time. Both methods employ a shared-parameter, teacher-student setup, with the teacher conditioned on PI during training. This framework allows effective RL and knowledge transfer without dependence on proprietary or inaccessible CoT traces.

Figure 1: Overview of the $\pi$-Distill framework. Training with PI-conditioned teacher and unconditioned student sharing parameters enables knowledge transfer to test-time policies that do not observe PI.

Methods

The $\pi$-Distill Algorithm

$\pi$-Distill consists of a parameter-shared teacher-student policy: the teacher receives PI and is trained to maximize expected reward, regularized by a reverse KL divergence towards the unconditioned student; the student is updated to distill the teacher’s behavior by imitating successful trajectories, also regularized by KL. The objective is a convex combination of teacher-centric ($\alpha=1$), student-centric ($\alpha=0$), or joint ($\alpha=0.5$) training, controlling the extent of knowledge transfer.

This coupled optimization avoids unstable off-policy distillation, as both teacher and student policies co-evolve, mitigating distribution shift and collapse. The framework generalizes to variational EM by treating the PI-conditioned teacher as an approximate posterior optimized towards a reward-weighted target, with the student distilling this posterior.

On-Policy Self-Distillation (OPSD)

OPSD adopts an on-policy approach whereby trajectories are sampled from the student policy, penalized by the reverse KL to the PI-conditioned teacher. This approach enables dense, per-token feedback and regularizes the student toward the teacher’s more informed policy, facilitating stable learning even with high-reward, sparsely-explored behaviors.

Experimental Setting and Benchmarks

Experiments are grounded in agentic tool-use environments including $\tau$-Bench (retail and airline domains), Travel Planner, and the GEM search-tool suite (covering seven out-of-domain datasets). Target models (Qwen3-4B, Qwen3-8B, and R1-Distill-Llama-8B) are non-frontier, demonstrating limited proficiency absent privileged signals. PI is mined from open-source DeepSeek-chat-v3.1 action traces and transformed into:

• Full tool calls (including arguments)
• Tool calls (function names only)
• Self-generated concise hints

This diversity in PI forms enables analysis of information density, distributional shift, and transferability.

Main Empirical Results

$\pi$-Distill demonstrates robust performance, consistently outperforming SFT+RL methods that assume full CoT access across nearly all experimental settings. With Qwen3-8B, $\pi$-Distill achieves relative improvements of up to 11.8% on Travel Planner, 2.08% on $\tau$-Bench retail, and 6.00% on $\tau$-Bench airline OOD, compared to SFT+RL with CoT. These gains are realized without requiring expensive checkpoint sweeps or privileged CoT supervision, requiring only a single-phase training process.

Further, OPSD matches or outperforms RL-based approaches lacking PI, especially as model scale increases. Importantly, both methods enable successful distillation even when models have previously seen PI during pretraining or SFT.

Out-of-Domain Generalization

Robustness of $\pi$-Distill and OPSD is validated on the GEM search-tool benchmarks (Figure 2). On Qwen3-8B, both methods substantially outperform the base models and standard RL, even surpassing SFT+RL with CoT in several OOD environments. Notably, standard off-policy RL shows significant degradation relative to the base model, while $\pi$-Distill and OPSD prevent such regressions.

Figure 2: OOD evaluation on the GEM suite demonstrates that $\pi$-Distill and OPSD generalize substantially better than RL and base models, with scaling benefits evident for larger models.

These findings underscore that explicit CoT supervision is less crucial for larger, more capable models, especially when leveraging action-only or summarized PI via on-policy training.

Analysis of Privileged Information Utility and Transfer Factors

The success of PI-based distillation is tightly linked to two factors:

• Initial KL divergence between teacher (PI-conditioned) and student (unconditioned) policies
• The utility of PI: empirical score improvement when conditioned on PI versus unconditioned rollout

Experiments (Figure 3 and Figure 4) reveal that joint teacher-student training($\alpha=0.5$) is the most robust setting, balancing exploitation and distributional regularization. Student-only ($\alpha=0$) training is effective when teacher-student divergence is low and PI utility is positive; teacher-only ($\alpha=1$) training collapses when the teacher learns to ignore PI or teacher-student collapse occurs. Notably, even when PI initially provides low utility, proper teacher training can enable significant student performance gain, demonstrating the importance of enabling the policy to learn to exploit inventive PI.

Figure 3: Final task performance is strongly correlated with initial teacher-student KL divergence, and the utility of PI; joint training yields notably stable improvements over independent variants.

Figure 4: Maximum attainable training gain $\Delta_{\text{max}}$ demonstrates that teacher-training enables policies to extract value from suboptimal PI.

For OPSD, the information richness of PI is the dominant predictor of performance, rather than initial KL divergence. Dense privileged signals, such as full tool call traces, consistently yield the best improvement (Figure 5).

Figure 5: OPSD performance is most sensitive to the density of informational content in the PI, with excessive KL only a secondary concern.

Regularization and Stability

The reverse-KL regularization controlled by $\beta$ is found to be essential for stabilizing training, especially when teacher updates are involved ($\alpha > 0$) (Figure 6). Ablation experiments show $\beta > 0$ is optimal in 14/18 $\pi$-Distill configurations.

Figure 6: Evaluation curves for $\pi$-Distill variants under varying $\beta$ show the necessity of nonzero regularization for peak performance and avoiding early collapse.

Theoretical Implications and Positioning

This work establishes practical and theoretical connections to latent-variable inference literature, demonstrating that the $\pi$-Distill update can be interpreted as an online, parameter-shared variant of variational EM with reward-driven optimization in place of likelihood. Unlike prior methods relying on sequential alternating EM or oracle targets—which are computationally brittle or require ground truth—the presented approach is simple, efficient, and deployable with arbitrary forms of PI, including action traces and synthesizable hints.

Practical and Future Implications

The ability to distill powerful agentic behavior using solely action traces or concise hints—without proprietary CoT—enables dissemination of frontier capabilities to open or smaller-scale LMs, democratizing access to complex agentic reasoning in high-stakes settings (e.g., tool use, planning, and dialog agents). Furthermore, the framework generalizes to other forms of privileged signals (retrieval results, structured scripts) and can be used to compress auxiliary behaviors into model weights for downstream deployment.

As models scale, results indicate that on-policy, PI-driven self-distillation will become increasingly effective relative to expensive SFT and RL methods that require explicit reasoning supervision. Future research may explore:

• Scalable distillation with broader PI sources, including user interaction logs or system-level annotations
• Robustness to distributional shift and adversarial PI
• Efficient adaptation in continual learning and OOD generalization
• Extensions to multimodal agentic domains

Conclusion

This work provides compelling evidence that privileged information distillation—operationalized via shared-parameter, joint teacher-student objectives—enables efficient and effective transfer of high-level agentic skills without reliance on Chain-of-Thought supervision. The empirical superiority and practical simplicity of $\pi$-Distill and OPSD recommend these algorithms as strong candidates for industry-scale distillation and rapid prototyping of agentic LMs, with broad implications for advancing model capabilities in domains where test-time PI is inaccessible.

Reference: "Privileged Information Distillation for LLMs" (2602.04942)