Group-level Dense Distillation for Cross-Modal Transfer

• The paper introduces GKD-Den, which achieves unpaired dense distillation by aligning activation-derived relation maps between NBI and WLI modalities.
• It leverages group-aware attention to model both local structural details and global semantic coherence without requiring image-level pairings.
• Empirical results reveal that GKD-Den outperforms traditional instance-level methods, improving AUC and demonstrating robust cross-modal performance.

Group-level Dense Distillation (GKD-Den) is a key module in the Pairing-free Group-level Knowledge Distillation (PaGKD) framework, designed for robust cross-modal knowledge transfer between Narrow-Band Imaging (NBI) and White-Light Imaging (WLI) modalities in gastrointestinal lesion classification. Unlike traditional distillation approaches that rely on precisely paired image instances, GKD-Den enables dense distillation at the group level using unpaired but class-consistent sets of images, achieving both global semantic alignment and local structural coherence without requiring image-to-image correspondence (Hu et al., 14 Jan 2026).

1. Motivation and Overview

Conventional instance-level dense distillation methods necessitate exact spatial matching between paired teacher-student feature maps, which is infeasible for cross-modal medical images due to significant intra-lesion variation in structure and appearance. GKD-Den addresses this limitation by operating on groups of images sharing the same lesion class label but lacking strict instance-wise pairing. Within each group, it aligns higher-order affinities and attentional patterns between modalities, enabling effective distillation in settings where paired data acquisition is impractical or cost-prohibitive.

2. Construction of Activation-Derived Relation Maps

For each modality $m \in \{\mathrm{NBI},\mathrm{WLI}\}$ and lesion class, a group of $K$ unpaired images is processed through the backbone network to obtain feature tensors $F^m \in \mathbb{R}^{K \times C \times H \times W}$. These are reshaped and channel-wise L2-normalized to yield flattened feature matrices $\bar F_k^m \in \mathbb{R}^{C \times S}$, with $S=H\times W$. All $K$ features are concatenated to form $\hat F^m \in \mathbb{R}^{C \times (K S)}$.

The affinity matrix is computed as $$A^m = (\hat F^m)^\top \hat F^m \in \mathbb{R}^{(K S) \times (K S)}$$. The activation-derived relation map is then obtained by applying a row-wise softmax with Transformer-style scaling:

$$[R^m]_{i,j} = \frac{\exp([A^m]_{i,j}/\sqrt{C})}{\sum_{j'=1}^{K S} \exp([A^m]_{i,j'}/\sqrt{C})}$$

This relation map encodes all-to-all patch affinities across the group and serves as a structural distillation target for the student network.

3. Group-Aware Attention and Relation-Guided Alignment

Group-level self-attention for each modality is established by projecting $\hat F^m$ into query, key, and value spaces via learned matrices $W_q, W_k, W_v \in \mathbb{R}^{d \times C}$, yielding $Q^m, K^m, V^m \in \mathbb{R}^{d \times (K S)}$. The attention logits are augmented with the log of the relation map:

$$\mathcal{L}^m = \frac{(Q^m)^\top K^m}{\sqrt{d}} + \alpha \log(R^m + \epsilon)$$

where $\alpha$ (typically $0.3$) moderates the influence of relation-guidance. The resulting group attention is

$$\mathrm{Attn}^m = \mathrm{softmax}(\mathcal{L}^m) V^m$$

reshaped back to $K \times d \times H \times W$ and, if desired, post-processed (e.g., with a $1 \times 1$ convolution and a residual connection).

This mechanism enables attention patterns—guided by class-wise affinities rather than direct correspondences—to be distilled between teacher and student networks.

4. Dense Distillation Loss Functions

GKD-Den enforces modality alignment via two complementary loss terms computed over each group:

• Relation-map consistency loss

$$\mathcal{L}_{\mathrm{rel}} = \frac{1}{(K S)^2} \left\| R^T - R^S \right\|_F^2$$

This term encourages the student’s patch affinity structure (WLI) to match the teacher’s (NBI).

• Attention alignment loss

$$\mathcal{L}_{\mathrm{attn}} = \frac{1}{K S} \left\| \mathrm{Attn}^T - \mathrm{Attn}^S \right\|_2^2$$

This aligns cross-patch attention patterns in the group-aware space.

The total dense distillation loss is

$$\mathcal{L}_{\mathrm{den}} = \lambda_{\mathrm{rel}} \, \mathcal{L}_{\mathrm{rel}} + \lambda_{\mathrm{attn}} \, \mathcal{L}_{\mathrm{attn}}$$

with $\lambda_{\mathrm{rel}} = 1.0$, $\lambda_{\mathrm{attn}} = 0.5$. This is combined with standard classification and prototype-level losses into the overall objective:

$$\mathcal{L} = \mathcal{L}_{\mathrm{cls}} + \lambda_{\mathrm{proto}}\, \mathcal{L}_{\mathrm{proto}} + \lambda_{\mathrm{den}}\,\mathcal{L}_{\mathrm{den}}$$

where $$\lambda_{\mathrm{proto}} = \lambda_{\mathrm{den}} = 1.0$$.

5. End-to-End Training Workflow

A training iteration for GKD-Den within PaGKD operates as follows:

1. For each lesion class, form groups of $K$ unpaired images per modality.
2. Extract group feature maps via forward pass through the modality-specific backbones.
3. Compute relation maps from activation affinities.
4. Compute relation-guided group attention for both teacher and student.
5. Calculate relation and attention consistency losses; aggregate with prototype and classification losses.
6. Perform backpropagation and parameter updates for the student network.

This procedure enables distillation without paired image data, leveraging only class labels and groupings.

6. Component Design Rationales

Relation maps facilitate local structural transfer by encoding fine-grained patch affinities within same-class groups. Aligning these maps compels the student to internalize the NBI teacher's discriminative structural signals, such as microvascular patterns. Group-aware attention alignment enforces global semantic consistency by ensuring the student attends to patch groups in ways congruent with the teacher’s focus. The use of group context—rather than instance-matched pairs—obviates the need for image-level alignment.

Because all computations are defined over the concatenation of group members, the method is inherently pairing-free: neither teacher nor student requires direct correspondences between sampled images.

7. Empirical Insights and Hyperparameters

Empirical results indicate group size $K=4$ optimally balances GPU memory demands and performance (Area Under Curve [AUC] peaks at $K=4$ among $K \in \{2,4,6\}$). The attention dimension is set at $d=64$ with a single attention head and a $1 \times 1$ convolution for post-attention feature fusion. Ablation studies on the PICCOLO dataset reveal that excluding GKD-Den from PaGKD reduces AUC by $1.5\%$, whereas excluding GKD-Pro yields a $0.9\%$ drop, and using traditional (paired, instance-level) dense distillation drops performance by $1.8\%$. These results validate the efficacy of group-level dense distillation in settings with unpaired, class-wise grouped data and its superiority for inducing both local and global modality-invariant representations (Hu et al., 14 Jan 2026).

Ablation Setting	AUC	Δ from Full PaGKD
Full PaGKD (GKD-Pro + GKD-Den)	0.926	0.0%
GKD-Pro only	0.911	–1.5%
GKD-Den only	0.917	–0.9%
Instance-level (paired)	0.908	–1.8%

A plausible implication is that, for cross-modal domains with substantial instance variance and limited paired data, GKD-Den provides a generic paradigm for robustly transferring dense spatial knowledge at the group level.

8. Significance and Applications

GKD-Den establishes a paradigm shift for cross-modal knowledge distillation, enabling robust transfer without the logistical and technical complexities of assembling paired datasets. By leveraging class-consistent, unpaired groupings and modeling both local and global correspondences through relation maps and group-aware attention, GKD-Den advances the state of the art in unpaired cross-modal learning for medical image classification and beyond (Hu et al., 14 Jan 2026). The resulting frameworks can more fully exploit vast clinical datasets, accelerating translation of diagnostic advantages between modalities.