Group-level Prototype Distillation (GKD-Pro)

• The paper introduces GKD-Pro, which distills teacher knowledge by aggregating group-level prototypes to bypass noisy instance-level features.
• It details methods for constructing prototypes and optimizing alignment and similarity losses across graph networks, object detection, and cross-modal imaging.
• Empirical evaluations show GKD-Pro consistently improves student model performance by robustly capturing global class characteristics.

Group-level Prototype Distillation (GKD-Pro) is a paradigm in knowledge distillation that leverages group-level, semantically robust representations—termed prototypes—to transfer information from a teacher network to a student network without relying on fine-grained object-level or sample-level pairings. By focusing on global or class-level characteristics, GKD-Pro offers robust mechanisms for distillation across diverse domains, including graph classification, object detection, and cross-modal image recognition. The approach is motivated by the observation that instance-level feature transfer is vulnerable to noise and modality discrepancy, whereas group-level prototypes encapsulate the stable, context-sensitive knowledge necessary for high performance.

1. Formal Definition and Prototype Construction

GKD-Pro operates on the principle of aggregating group-level features from a cohort of instances within the same semantic class to form prototypes. In graph node classification tasks, such as described in PGKD (Wu et al., 2023), the prototype for class $c$ is computed as the average of the teacher embeddings:

$$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$

where $h_i^T$ is the teacher's embedding of node $i$.

In cross-modal medical image classification (PaGKD (Hu et al., 14 Jan 2026)), sets of images from both modalities are sampled for each class $c$. Let $G_W^c$ and $G_N^c$ be the groups of White-Light Imaging (WLI) and Narrow-Band Imaging (NBI) images, respectively. Global feature maps $F(x)$ are pooled, and K learnable queries are used to produce K semantic prototypes for each modality:

$\alpha_{m,k}^c = \text{softmax}( Q_k \cdot \overline{F}_m^c^T / \sqrt{D} ), \ P_{m,k}^c = \alpha_{m,k}^c \cdot \overline{F}_m^c,$

where $\overline{F}_m^c$ concatenates all spatial features from modality $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$0 in group $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$1, and $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$2 is the $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$3-th query vector.

In object detection (Tang et al., 2022), prototypes are selected via a prototype generation module (PGM) that identifies a dictionary of K basis vectors in the feature space. Instance features are projected onto these prototypes, yielding low-dimensional coefficient vectors that serve as global representations for distillation.

2. Distillation Loss Functions

GKD-Pro employs losses that enforce both alignment of student instance embeddings with teacher prototypes and similarity of student/teacher prototypes.

• Graph node classification (Wu et al., 2023) introduces:
  • Alignment (Structure-distortion) Loss: $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$4
  • Prototype-similarity Loss: $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$5
• Cross-modal image classification (Hu et al., 14 Jan 2026) employs:
  • Classification Loss $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$6
  • Prototype Alignment Loss: $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$7
  • The total GKD-Pro loss: $$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$8
• Object detection (Tang et al., 2022) includes global-alignment, local-feature, and response-based losses weighted by instance reliability:

$$p_c^T = \frac{1}{|\mathcal{V}_c|} \sum_{i \in \mathcal{V}_c} h_i^T,$$9

where $h_i^T$0 are the K-prototype coefficients for the teacher/student, and reliability weight $h_i^T$1 suppresses noisy transfers.

3. Training Algorithms and Integration

GKD-Pro is implemented via minibatch-wise operations, combining prototype computation, forward passes, loss evaluation, and parameter updates. A prototypical pass in PGKD (Wu et al., 2023) involves:

1. Obtaining teacher embeddings for the batch.
2. Updating teacher prototypes either by recomputation or momentum averaging.
3. Computing student embeddings and logits.
4. Calculating all relevant losses.
5. Backpropagation and parameter updates.
6. Momentum updates of student prototypes (optional).

In cross-modal PaGKD (Hu et al., 14 Jan 2026), GKD-Pro integrates as follows (Editor’s term: “semantic query pooling”):

• Sample two unpaired groups with the same lesion class.
• Extract feature maps from backbone.
• Compute group-level prototypes via learned queries.
• Evaluate classification and prototype losses.
• Update parameters by standard optimization.

In object detection (Tang et al., 2022), prototype selection is accomplished by greedy matching-pursuit, followed by projection and loss computation in prototype space.

4. Empirical Performance and Ablation Studies

GKD-Pro demonstrates consistent improvements across domains and benchmarks:

Benchmark	Teacher	Student Baseline	Instance-level KD	GKD-Pro
Cora	GNN	74.2% (MLP)	77.5% (KD)	81.3%
Citeseer	GNN	71.0% (MLP)	73.8% (KD)	77.0%
Pubmed	GNN	79.8% (MLP)	82.1% (KD)	84.7%
COCO (Det)	Faster RCNN	39.8 mAP	38.4 mAP	40.6 mAP
VOC (Det)	Faster RCNN	56.3 mAP	54.2 mAP	56.7 mAP
Medical AUC	WLI Baseline	—	—	+1.8% to +3.3%

Ablations indicate distinct contributions of alignment and similarity losses (e.g., removal of $h_i^T$2 drops Cora accuracy by 2.2 points; removal of $h_i^T$3 drops by 0.9 points (Wu et al., 2023)). In cross-modal medical imaging, disabling prototype alignment yields AUC reductions of up to 3.1% (Hu et al., 14 Jan 2026).

Combination with instance-level methods in detection yields additive improvements (+0.3 to +0.5 mAP) (Tang et al., 2022), supporting the complementarity of prototype- and instance-level knowledge.

5. Theoretical and Practical Justification

The primary rationale for GKD-Pro is to mitigate noise and instability present in instance-level feature transfer, particularly prevalent in settings with small, occluded, or modality-divergent samples (Tang et al., 2022, Hu et al., 14 Jan 2026). Prototypes capture the essential, context-averaged structure of a class or group, enabling the student to mimic the teacher’s core subspace rather than volatile instance features. This approach is robust to missing structural information (e.g., absent graph edges (Wu et al., 2023) or lack of image pairing (Hu et al., 14 Jan 2026)) and scales to cross-modal and cross-architecture settings.

Instance reliability is intrinsically managed via prototype coefficient discrepancy, which down-weights unreliable (noisy) examples (Tang et al., 2022). Shared queries in cross-modal GKD-Pro enforce modality-invariant feature summarization, driving semantic consistency and facilitating application to unpaired data (Hu et al., 14 Jan 2026).

6. Applications and Extensions

GKD-Pro has been applied in:

• Graph neural net distillation: Edge-free transfer of GNN knowledge into MLPs, achieving near-teacher performance without graph convolution (Wu et al., 2023).
• Object detection distillation: Robust distillation from strong, noisy teachers to compact students, outperforming both instance-level and prior global methods (Tang et al., 2022).
• Cross-modal medical image classification: Effective knowledge transfer between modalities (NBI→WLI) using unpaired data, with semantic consistency across modalities via query-based prototypes (Hu et al., 14 Jan 2026).

The paradigm is readily extensible to CNN/Transformer backbones and can be combined with local-instance distillation for enhanced performance.

7. Comparison with Related Methods and Limitations

Most prior distillation frameworks transfer instance-level knowledge: either via direct feature mimicry, relation matching, or response-based losses (e.g. [FGFI], [FBKD], [RKD], [GID], [DeFeat], [Hinton]). These approaches suffer in noisy scenarios, where teacher feature quality is low or cross-modal discrepancies exist.

GKD-Pro’s “space alignment” via prototypes is less susceptible to sample-level noise, allows groupwise summarization, and can filter unreliable transfers using coefficient discrepancy (Tang et al., 2022). This methodology is complementary to instance-level methods and consistently provides additive performance gains.

Limitations include potential loss of fine-grained instance specificity and dependence on representative group formation. Careful balancing of prototype and local losses is necessary to avoid excessive averaging, which may obscure rare but relevant features.

In summary, Group-level Prototype Distillation enforces robust, global alignment between teacher and student by distilling class- or group-level prototypes, yielding reliable knowledge transfer benefitting diverse tasks including graphs, detection, and cross-modal learning (Wu et al., 2023, Hu et al., 14 Jan 2026, Tang et al., 2022).