SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction

Abstract: Most existing knowledge distillation methods focus on homogeneous models (e.g., CNN-to-CNN), thereby overlooking the flexibility and potential of knowledge transfer across heterogeneous models. Due to intrinsic inductive bias discrepancies between heterogeneous models that cause spatial distribution inconsistencies, prior heterogeneous distillation methods often weaken or discard spatial information in heterogeneous representations. However, the spatial information in representations often encodes transferable global structural semantics as well as architecture-specific local details, and therefore should not be directly ignored. To better leverage the spatial information encoded in heterogeneous representations, we propose a Spatial-Frequency Joint-Aware Heterogeneous Knowledge Distillation framework (SFKD). By leveraging the complementary properties of wavelet transform spatial locality and Fourier representations in characterizing global energy distributions, we first apply multi-level discrete wavelet transform to explicitly decouple spatial information. The resulting wavelet sub-bands are further refined by a dual-stream dual-stage refinement module, and finally combined with a Gaussian-filtered frequency loss to selectively capture informative global information. Extensive experiments on multiple benchmark datasets under both homogeneous and heterogeneous models demonstrate the superiority of our method.

• The paper presents a novel spatial-frequency joint-aware method that decouples global and local feature representations using multi-level wavelet transforms.
• It introduces a dual-stream dual-stage spectral refinement module, combining convolution and transformer-based strategies to enhance feature alignment.
• Empirical evaluations on CIFAR-100 and ImageNet-1K demonstrate significant accuracy improvements over traditional heterogeneous distillation methods.

SFKD: A Spatial–Frequency Joint-Aware Framework for Heterogeneous Knowledge Distillation

Introduction

The paper "SFKD: Spatial--Frequency Joint-Aware Heterogeneous Knowledge Distillation via Multi-Level Wavelet Spectral Interaction" (2607.01906) introduces a principled framework for knowledge distillation (KD) across heterogeneous deep neural network architectures. In contrast to traditional KD methods, which are predominantly tailored for homogeneous model pairs (e.g., CNN-to-CNN), SFKD targets the challenging setting of transferring knowledge between teacher and student networks with divergent inductive biases (e.g., Transformer-to-CNN, MLP-to-Transformer). This work addresses core limitations of prior heterogeneous KD paradigms, specifically the inadequate handling of spatial information embedded in intermediate representations—information that is often suppressed or ignored due to significant feature distribution discrepancies across architectures.

SFKD presents a unified spatial–frequency approach leveraging complementary properties of multi-level discrete wavelet transform (MDWT) and Fast Fourier Transform (FFT), in conjunction with a dual-stream dual-stage spectral refinement module (DS²SR) and a Gaussian-filtered frequency-domain contrastive loss (GFFL). This framework facilitates fine-grained decoupling, selective refinement, and effective alignment of both global and local semantic content in features, enabling more efficient and robust heterogeneous KD.

Motivation and Background

Contemporary networks such as CNNs, Transformers, and MLP-based backbones embody fundamentally distinct priors—locality and translation invariance vs. long-range dependency modeling—which manifest as substantial inconsistencies in their learned spatial features. Prior heterogeneous KD approaches frequently resort to compressing intermediate features into spatially agnostic embeddings or logit spaces to avoid spatial distribution mismatches, thereby discarding a considerable amount of transferable structural information. This suppression, although mitigating the representational gap, also precludes the student from exploiting critical global and architecture-specific local semantics, resulting in suboptimal knowledge transfer.

The empirical analysis in the paper highlights that teacher models with different architectures from the student sometimes lead to superior distilled models, demonstrating the underutilized potential of cross-architecture KD. However, the challenge lies in bridging the spatial-feature gap without over-suppressing informative spatial cues.

Methodology

Multi-Level Discrete Wavelet Transform (MDWT)

SFKD first employs a multi-level Haar wavelet decomposition on teacher and student feature maps, recursively partitioning them into low-frequency (LF) and high-frequency (HF) sub-bands. LF sub-bands capture global structures, while HF components encode local, architecture-specific details such as edges and textures. This explicit decoupling transforms the complex spatial domain problem into a set of structured sub-bands amenable to targeted processing and alignment.

Figure 1: Heterogeneous feature maps, multi-level wavelet sub-bands via MDWT, and their FFT spectral features, illustrating how spectral transformation reduces spatial discrepancies.

Dual-Stream Dual-Stage Spectral Refinement (DS²SR) Module

Given the limited representational capacity of typical student architectures, student sub-bands are prone to higher spectral instability and residual noise, complicating sub-band-level feature alignment. The DS²SR module addresses this by refining the student's wavelet sub-bands in parallel through two complementary streams:

• Local Convolutional Refinement (LCR): Convolutional blocks (ResConvBlock) enhance fine-grained local details.
• Global Transformer Refinement (GTR): Transformer-based blocks model long-range dependencies and global semantics, employing a novel 4D positional encoding that accounts for wavelet level, sub-band index, and spatial locality.

The outputs of LCR and GTR are further integrated by adaptive weighted fusion (AWF) modules, separately for LF and HF sub-bands.

Figure 2: Comparison of conventional heterogeneous distillation vs. SFKD, and detailed structure of the DS²SR module with separate LCR and GTR streams and their fusion.

Figure 3: Detailed ResConvBlock architecture used within the LCR stream.

Gaussian-Filtered Frequency Loss (GFFL)

To further regularize sub-band alignment and circumvent remaining spatial distribution mismatches, the framework operates in the frequency domain. The MDWT-decomposed sub-bands are transformed via FFT. Amplitude spectra are extracted, and sub-band-specific Gaussian filters (low-pass for LF, high-pass for HF) are applied. Alignment between filtered student and teacher spectra is encouraged using an InfoNCE-based contrastive loss. This dual domain (spatial + frequency) alignment enables SFKD to simultaneously emphasize transferable global and local knowledge.

Additionally, the refined student sub-bands are reconstructed via inverse MDWT, and the recovered spatial features are aligned with the original teacher using a non-filtered GFFL loss and supervised via Kullback-Leibler divergence between logits.

Figure 4: (a) 4D positional encoding for tokens, (b) Gaussian low-pass and high-pass frequency masks, (c) attention mask delineating allowed cross-frequency interactions in GTR.

Experimental Evaluation

SFKD is evaluated on CIFAR-100 and ImageNet-1K across numerous heterogeneous and homogeneous teacher-student pairs, including CNNs (ResNet, MobileNet, ConvNeXt), Transformers (Swin, ViT, DeiT), and MLP-based models (MLP-Mixer, ResMLP). Benchmarks against conventional logit-based, feature-based, recent heterogeneous-specific KD methods, and strong homogeneous baselines are provided. Key results include:

• Average Top-1 accuracy improvements of 10.92% (CIFAR-100) and 2.51% (ImageNet-1K) over training from scratch.
• SFKD consistently surpasses the state-of-the-art FBT method by margins of 2.54% (CIFAR-100) and 0.20% (ImageNet-1K) in feature-based distillation.
• In homogeneous settings, SFKD also yields the best results (e.g., ResNet34→ResNet18: 72.34%, outperforming strong attention-based and review-based distillation).
• Ablation studies affirm the necessity and complementary benefits of both LCR and GTR streams within DS²SR, and show that the GFFL provides superior performance over standard MSE or InfoNCE in the frequency domain.

The computational efficiency of SFKD remains competitive, with training/inference costs comparable to or better than baseline distillation methods, due to judicious module design and reliance on final-layer feature maps only.

Implications and Future Directions

SFKD demonstrates that the spatial information embedded in heterogeneous representations is transferable when appropriately decoupled and selectively aligned in both spatial and frequency domains. This challenges the prevalent paradigm of suppressing spatial cues for heterogeneous KD and provides a pathway to distillation strategies that can be generalized to arbitrary source and target architectures.

Practically, this unlocks broader flexibility in teacher selection and better interoperability among model families, which is vital for deployment in diverse or resource-constrained environments. Theoretically, SFKD’s architecture suggests further investigation into principled multi-domain representation alignment, hybrid signal decomposition techniques (e.g., learned wavelet transforms), and more parameter-efficient refinement modules.

Potential future work includes exploring lightweight or dynamic refinement modules to reduce redundancy, tighter integration of learned filters (both wavelet and frequency-domain), and extensions to other tasks such as segmentation or detection where spatial detail is critical.

Conclusion

SFKD introduces a comprehensive spatial–frequency-aware framework for heterogeneous knowledge distillation that marries wavelet feature decoupling, dual-stream refinement, and frequency-domain contrastive alignment. The approach demonstrates consistent numerical superiority across a wide spectrum of architectures and dataset scales without incurring excessive computational overhead. SFKD establishes that preserving and exploiting structured spatial information in intermediate representations is critical for effective cross-architecture knowledge transfer, paving the way for more universal and powerful distillation strategies.