Logits-Level Distillation (LDist)

• Logits-Level Distillation (LDist) is a method that directly aligns teacher and student logits to transfer complete output distributions.
• It leverages KL divergence with temperature scaling to effectively soften predictions, making it ideal for high-dimensional outputs and capacity discrepancies.
• Recent extensions include grouped logits, ranking-based losses, and adaptive temperature schemes to boost performance in vision, language, and audio domains.

Logits-Level Distillation (LDist) is a knowledge distillation methodology that operates by directly aligning the output logits of a teacher model with those of a student. The primary objective is to facilitate the transfer of both class-relevant and relational information, leveraging the full structure of the output distribution, rather than relying solely on hard labels. LDist is particularly advantageous in contexts with high-dimensional output spaces or where model capacity discrepancies preclude naïve feature-level alignment. It has been extended and refined across diverse application domains, including vision, language, audio, and cross-modal models.

1. Mathematical Foundations and Core Formulation

At its core, LDist minimizes a divergence—most commonly the Kullback-Leibler (KL) divergence—between the teacher and student softmax-normalized logits, potentially after temperature scaling. For an input $x$, teacher logits $z^{(T)} \in \mathbb{R}^V$ and student logits $z^{(S)} \in \mathbb{R}^V$, temperature $t > 0$, the softened class probabilities are: $$P^{(T)}(k) = \frac{\exp(z^{(T)}_k / t)}{\sum_{j=1}^V \exp(z^{(T)}_j / t)}$$

$$P^{(S)}(k) = \frac{\exp(z^{(S)}_k / t)}{\sum_{j=1}^V \exp(z^{(S)}_j / t)}$$

The standard LDist loss is then: $$\mathcal{L}_{\text{LDist}} = \sum_{k=1}^{V} P^{(T)}(k) \log\frac{P^{(T)}(k)}{P^{(S)}(k)}$$ The total training loss typically augments this with a cross-entropy term on ground-truth labels, weighted by tuning parameters. Variations on this basic theme adapt the loss to different structure, class grouping, or token regimes (Yang et al., 2 Feb 2026Zhao et al., 2023).

In sequence models, token-wise LDist sums over sequence positions, and supports per-token temperature and selective focus via learned indicators or difficulty metrics (Xie et al., 13 Oct 2025).

2. Extensions and Methodological Innovations

LDist has been actively developed to address key limitations—such as overemphasis on noisy logit tails, lack of relational structure transfer, and difficulties in high-class-count regimes. Notable extensions include:

• Grouped Logits Partitioning: Partitioning the logit vector into primary (informative) and secondary (tail/noise) groups, only transferring the former and using separate binary KL for marginal mass (Zhao et al., 2023), yielding strong gains in face recognition with large class counts.
• List-wise and Ranking-based Losses: Plackett-Luce Distillation (PLD) imposes full teacher-derived class rankings, integrating the ranking with position-specific confidence weights in a convex surrogate loss (Bassam et al., 14 Jun 2025). This subsumes and generalizes cross-entropy, showing robustness and accuracy improvement.
• Logit Difference and Top-K Methods: Bi-directional Logits Difference (BiLD) loss aligns pairwise differences among top-k teacher and student logits, encoding internal ranking structure and filtering long-tail noise, critical for LLMs (Li et al., 2024). TopKD applies scaling and decoupled cosine losses to explicitly emphasize high-value logits (Wang et al., 6 Aug 2025).
• Contrastive and Semantic Geometry Losses: Multi-perspective contrastive approaches build instance-wise, sample-wise, and category-wise InfoNCE losses over logits, thereby aligning not only probability mass but geometric relations (Wang et al., 2024). MCLD in vision tasks demonstrates that logit geometry alignment yields marked improvements over classical KL-based distillation.
• Adaptive and Token-wise Temperature: Rather than fixing a global temperature, several methods adapt the temperature per-sample or per-token, based on logit spread, uncertainty, or difficulty metrics. Example: AdaKD learns per-token difficulty via Hellinger distance and assigns inverse-scaling temperature, focusing gradients on hard tokens (Xie et al., 13 Oct 2025); maximal-logit-based adaptive temperature enforces convergence to logit correlation (Matsuyama et al., 12 Mar 2025).
• Structural and Transport-based Losses: In scenarios with vocabulary mismatch (e.g., cross-tokenizer LLMs), Universal Logit Distillation (ULD) replaces KL with a Wasserstein-1 distance between ordered softmax distributions, supporting alignment across non-overlapping vocabularies (Boizard et al., 2024). Graph-on-Logits Distillation (GLD) constructs co-activation graphs over logit dimensions and aligns their (compressed) structure via Gromov–Wasserstein distance (Wang et al., 20 May 2025).
• Hybrid and Auxiliary-Head Architectures: Empirical analysis has shown that combining probability-level (softmax) and logit-level (pre-activation) losses in a single head can destabilize learning, particularly at the linear classifier; splitting into dual heads mitigates classifier collapse and enables joint utilization (Yang et al., 2024).

3. Application Domains and Architectures

LDist is not restricted to conventional classification, but is deployed in:

• Audio-LLMs: Distilling multi-segment token groups (audio, response, prompt) in LALMs for emotion recognition, using LDist only on select segments due to differing roles (Yang et al., 2 Feb 2026).
• Multi-label Classification: Extending LDist via per-class binary sigmoid/KL for multi-label tasks rather than entire-softmax, directly aligning independent one-versus-all outputs (Yang et al., 2023).
• Object Detection: Incorporating localization logits (bounding box distributions) via cross-entropy on discretized edge bins, enabling transfer of teacher's spatial uncertainty (Zheng et al., 2022).
• LLMs and Cross-Modal Models: Token-adaptive, ranking-sensitive, and optimal-transport variants of LDist are now common in LLM distillation and fusion (Li et al., 2024Xie et al., 13 Oct 2025Wang et al., 20 May 2025).

A representative table summarizing several leading LDist methodologies:

Method	Domain/Use Case	Key Innovation
PL-Distill (Yang et al., 2 Feb 2026)	SER in LALMs	Multi-segment KL, selective loss
GKD (Zhao et al., 2023)	Face recognition	Primary/secondary logit partitioning
PLD (Bassam et al., 14 Jun 2025)	Vision (ImageNet)	List-wise Plackett-Luce loss
BiLD (Li et al., 2024)	LLMs (NLP)	Top-k logit differences, bidirectional
TopKD (Wang et al., 6 Aug 2025)	Vision, transfer	Top-K scaling and cosine decoupling
AdaKD (Xie et al., 13 Oct 2025)	LLMs	Token-adaptive focusing, inverse temp
ULD (Boizard et al., 2024)	LLMs, cross-tokenizer	Wasserstein OT alignment
GLD (Wang et al., 20 May 2025)	LLM fusion	Graph-based co-activation alignment

4. Practical Implementation Considerations

Central parameters in LDist implementation include:

• Temperature: Crucial for softening distributions and controlling gradient sensitivity. Adaptive schemes (per-sample/token) now outperform fixed-temperature baselines (Xie et al., 13 Oct 2025Matsuyama et al., 12 Mar 2025).
• Group Selection: For very large class counts, strategies to partition or sparsify the logit vector (top-K, grouped, mask) are essential to avoid overfitting to noisy tails (Zhao et al., 2023Li et al., 2024Wang et al., 6 Aug 2025).
• Class and Batch Normalization: Several successful methods employ Z-score or perception-based standardization of logits (per-class mean/variance) to ensure capacity-aligned transfer and stabilize optimization (Sun et al., 2024Hossain et al., 2023).
• Loss Weighting: Balancing the strengths of cross-entropy, KL (entire or partial logits), and auxiliary latent geometry losses via manual or data-driven grid-search remains standard.
• Architectural Decoupling: Where multiple head structures are used (e.g., dual-head KD), gradients are routed separately for stability (Yang et al., 2024).

5. Empirical Impact and Benchmarking

Across image, audio, and language domains, LDist methods consistently outperform vanilla KL-based distillation and even many advanced feature-based approaches in challenging scenarios. Key quantitative highlights include:

• On SER (IEMOCAP/RAVDESS/SAVEE), LDist alone achieves up to +3% UA gain over standard KL, and further +3–8% when coupled with projector-level distillation (Yang et al., 2 Feb 2026).
• In large-class-count vision tasks, Grouped KD improves LFW-style and MegaFace student accuracy by +3–7%, and TPR by +4–7% in low-FPR regimes (Zhao et al., 2023).
• On ImageNet, PLD yields +0.4–1.1 pp Top-1 over KD, and maintains robustness to optimizer and training schedules (Bassam et al., 14 Jun 2025).
• In LLMs, BiLD outperforms vanilla KL by up to +3.5% on Qwen and +1.1% on BLOOM across 13 datasets (Li et al., 2024); AdaKD lifts ROUGE-L by +1.3 vs RKD with negligible compute overhead (Xie et al., 13 Oct 2025).
• On fusion tasks, GLD provides +2.3 pp over the best non-graph baseline across 11 reasoning, coding, and math benchmarks (Wang et al., 20 May 2025).

6. Limitations, Defenses, and Future Directions

LDist is sensitive to:

• Support mismatch: Classical KL-based LDist cannot operate when teacher and student vocabularies do not match; ULD (OT-based) addresses this limitation (Boizard et al., 2024).
• Overfitting to noisy components: Matching full logit vectors transfers uninformative or unstable tail elements, motivating top-k filtering and contrastive geometry methods (Li et al., 2024Wang et al., 6 Aug 2025Wang et al., 2024).
• Model extraction vulnerabilities: LDist exposes models to extraction attacks when full logit information is released; minimizing the conditional mutual information $I(X;Z|Y)$ via logit post-processing provides an information-theoretic defense without degrading task accuracy (Fang et al., 3 Feb 2026).

Further avenues include:

• Extending token-adaptive and structural loss variants to cross-modal and multi-task architectures;
• Automated curriculum and weighting schemes for selective distillation focus;
• Efficient alignment under vocabulary growth and distribution shift;
• Theoretical analyses of information transfer efficiency and resistance to adversarial extraction.

LDist thus remains a central, rapidly-evolving mechanism in knowledge distillation, driving both practical model compression and theoretical understanding of teacher-student generalization in deep learning.