High-Frequency Feature Injection in ML

• High-frequency feature injection refers to techniques that separate and emphasize components like edges, textures, and oscillations to counter spectral bias in ML models.
• It employs methods such as wavelet and Fourier decompositions, spatial filtering, and regularization to enhance robustness and generalization across applications like image editing and PDE solvers.
• Empirical results show marked improvements, including higher LPIPS/SSIM in image editing and increased accuracy in adversarial settings, validating its impact on model performance.

High-frequency feature injection is a set of techniques designed to counteract the loss or underutilization of high-frequency information—such as sharp edges, textures, and localized oscillations—in machine learning models. This class of methods has emerged in response to three broad phenomena: (1) the spectral bias in deep and shallow networks towards low-frequency solutions, (2) adversarial vulnerabilities and texture overfitting in convolutional architectures, and (3) the inherent low-pass filtering behavior in spiking or biologically inspired models. High-frequency feature injection intervenes at various points in the data or network pipeline, using explicit frequency decomposition, architectural modifications, or regularization schemes, thereby enhancing fidelity, robustness, or generalization in applications ranging from image editing and adversarial defense to partial differential equation (PDE) solvers and neuromorphic vision.

1. Mathematical Foundations and Decomposition

High-frequency feature injection relies on explicitly separating the frequency content of activations, images, or latent representations, followed by selective manipulation or preservation of high-frequency components.

• Wavelet Decomposition: Given an image or feature tensor $x$, discrete wavelet transform (DWT) is commonly employed to split the input into subbands representing low- and high-frequency content. Single or multi-level DWT produces coefficients

$\{LL, LH, HL, HH\} = \mathrm{DWT}(x)$

where $LL$ encodes coarse structure (low-frequency), and $LH, HL, HH$ contain vertical, horizontal, and diagonal high-frequency detail (Anshumaan et al., 2020, Liao et al., 1 Dec 2025).

• Fourier Decomposition: In the frequency domain, the discrete Fourier transform (DFT) of a feature map $\mathcal{X}$ yields

$$\widehat{\mathcal{X}}(u, v) = \mathcal{F}\{\mathcal{X}\}(u, v)$$

and frequency masking functions $M_{LF}$, $M_{HF} = 1 - M_{LF}$ define low- and high-frequency regions. The inverse transform isolates the respective frequency bands (Zhang et al., 2024, Ren et al., 14 Oct 2025).

• SRM Filtering: In adversarial robustness, Spatial Rich Model (SRM) high-pass filter banks $\{\kappa_k\}$ are applied channel-wise to feature maps, with the output representing $\mathcal{X}_{HF} = \sum_k \kappa_k * \mathcal{X}$, and $\mathcal{X}_{LF} = \mathcal{X} - \mathcal{X}_{HF}$ (Zhang et al., 2024).

2. Algorithms and Architectures for High-Frequency Injection

a) Adaptive Injection in Image Editing

FreqEdit (Liao et al., 1 Dec 2025) exemplifies high-frequency feature injection for preserving detail across multiple iterative image edits. The approach operates in latent velocity space for diffusion-based editors, employing:

• Reference Extraction: The mean “reference velocity” $v^{\rm ref}$ from the clean context image is computed as

$$v^{\rm ref}_{t_i} = \frac{Z^{\rm ref}_0 - Z_{t_i}}{t_N - t_i}$$

• Wavelet Injection: Two-level DWT is applied to both editing and reference velocities. High-frequency detail subbands ($\mathbf{D}^{(1)}, \mathbf{D}^{(2)}$) are adaptively fused via

$$\tilde{\mathbf{D}}^{(\ell)}(x) = \mathbf{D}_{\rm edit}^{(\ell)}(x) + \boldsymbol{\alpha}^{(\ell)}(x) \odot [\mathbf{D}_{\rm ref}^{(\ell)}(x) - \mathbf{D}_{\rm edit}^{(\ell)}(x)]$$

where $\boldsymbol{\alpha}(x)$ is a spatially varying weight computed from velocity differences.

• Path Compensation: To prevent the progressive locking of spurious reference features, a “compensation” step periodically removes the integrated difference between the corrected and original editing velocities.

This ensures continual reintroduction of fine structure without impeding semantic edits or accumulating artifacts.

b) Frequency-Aware Adversarial Robustness

The High-Frequency Feature Disentanglement and Recalibration (HFDR) module (Zhang et al., 2024) mitigates the low-frequency bias induced by adversarial training:

• Feature Disentanglement: Frequency-specific attention maps $A_{HF}, A_{LF}$ (with $A_{LF}=1-A_{HF}$) are derived via Gumbel-Softmax over high- and low-frequency activations, enabling channel- and spatially aware weighting:

$$f_{HF} = \mathcal{X} \odot A_{HF}, \qquad f_{LF} = \mathcal{X} \odot A_{LF}$$

• Recalibration Block: High-frequency features are passed through a recalibration network $\varphi(\cdot)$, yielding $\tilde f_{HF}$. Final fusion gives

$\tilde f = \tilde f_{HF} + f_{LF}$

• Regularization: Frequency Attention Regularization (FAR) constrains the ratio of high-frequency attention $\|A_{HF}\|/\|1-A_{HF}\|$ to a preset target $\beta$, promoting balanced extraction.

c) Fourier Feature Injection in Physics-Informed Solvers

General Fourier feature physics-informed ELM (GFF-PIELM) (Ren et al., 14 Oct 2025) addresses the spectral bias in scientific ML:

• Fourier Activation: Each hidden neuron computes a feature

$\phi_m(x, t) = \cos(\omega_m^\top [x, t] + b_m)$

with neuron-specific frequency coefficients $\omega_m$ spanning target bands.

• Adaptive Initialization: The frequency range $[\delta_1, \delta_M]$ is set by monitoring the output weights $\beta_m$ after a trial solve, and narrowing to the most relevant band.

This ensures high-frequency (multi-scale, oscillatory) components of PDE solutions are representable without excessive depth or complexity.

d) Operator-Based High-Frequency Restoration in Spiking Transformers

In spiking transformer architectures, where inherent neuronal low-pass filtering suppresses high-frequency features, Max-Former (Fang et al., 24 May 2025) employs:

• Max-Pooling in Patch Embedding: In addition to standard embedding, max-pooling is injected to amplify local intensity peaks—functionally a high-pass enhancement.
• Small-Kernel Depth-Wise Convolution (DWC): Early token mixing is implemented using DWC, keeping mid- and high-frequency content more intact compared to spiking self-attention or low-pass pooling.

These interventions directly counter the bandwidth narrowing in multilayer spiking systems.

3. Adversarial Manipulation via High-Frequency Perturbation

WaveTransform (Anshumaan et al., 2020) demonstrates that perturbations targeted at high-frequency subbands are particularly effective in crafting transferable adversarial examples. The procedure:

• Selects High-Frequency Wavelet Bands: Perturbations $\delta_{LH}, \delta_{HL}, \delta_{HH}$ are optimized in the wavelet domain, reconstructing adversarial images via inverse DWT.
• Gradient-Based Optimization: Iterative projected sign-gradient steps are applied to maximize loss on the perturbed image, respecting an $\ell_\infty$ norm constraint.

Empirically, high-only attacks drastically reduce classification accuracy even under strong defenses, with near-perfect transferability across architectures and minimal perceptual distortion.

4. Experimental Impact and Performance Gains

High-frequency feature injection yields quantifiable performance improvements and robustness, as evidenced by multiple studies:

Domain	Method/Model	Impact/Results
Image Editing	FreqEdit (Liao et al., 1 Dec 2025)	Enables >10 successive edits with preserved quality; raises LPIPS/SSIM by 5–15 points over 7 SOTA baselines
Adversarial Robustness	HFDR (Zhang et al., 2024)	Improves PGD-10 white-box accuracy (CIFAR-10, WRN34-10) by +2.6% (from 56.1% to 58.7%)
PDE/SciML	GFF-PIELM (Ren et al., 14 Oct 2025)	Reduces L2 error by up to 10^4× for variable- and high-frequency PDEs; matches vanilla ELM in efficiency
Spiking Transformers	Max-Former (Fang et al., 24 May 2025)	Boosts ImageNet top-1 accuracy from 74.81% to 82.39% (+7.58%) at lower energy cost; ablations confirm additivity
Adversarial Attack	WaveTransform (Anshumaan et al., 2020)	Lowers accuracy on adversarial CIFAR-10 (ResNet-50) to 1.03% ("high-only"), exhibits strong transferability

5. Theoretical and Biological Motivation

Spectral analysis of neural architectures reveals strong low-pass tendencies across several domains:

• Spiking neural networks: Membrane dynamics (e.g., LIF neuron) mathematically equate to IIR low-pass filters, attenuating high-frequency signals layer-wise (Fang et al., 24 May 2025).
• Standard convolutional nets: Texture and edge detectors in early layers are sensitive to high-frequency perturbations, explaining the potency of such attacks (Anshumaan et al., 2020).
• PINN and ELM solvers: Activation functions such as tanh struggle to represent rapidly oscillating solutions, requiring alternative bases or explicit feature injection (Ren et al., 14 Oct 2025).

These insights justify explicit intervention in the frequency domain to retain or restore essential information for downstream performance and robustness.

6. Limitations and Open Challenges

While high-frequency feature injection confers substantial benefits, several challenges persist:

• Manual selection of pooling/convolution parameters in architectural approaches (e.g., Max-Former) may limit automatic adaptation across tasks (Fang et al., 24 May 2025).
• Regularization schemes must carefully balance high- and low-frequency contributions to avoid overfitting noise or introducing visual artifacts (Zhang et al., 2024, Liao et al., 1 Dec 2025).
• Bandwidth injection in PDE solvers requires domain knowledge when initializing frequency parameters, although adaptive strategies alleviate this (Ren et al., 14 Oct 2025).

A plausible implication is that future methods will move toward learnable or data-driven frequency decomposition, extending the flexibility and generality of current injection strategies. Further, exploration of cross-domain benefits—such as event-based sensing, audio, and structural health monitoring—remains active.

7. Broader Implications and Future Directions

The increasing use of high-frequency feature injection highlights several broader implications:

• Network design: Explicit frequency-aware components may be integrated into layer design, blending architectures from signal processing, neuroscience, and deep learning.
• Generalization and robustness: Frequency-balanced representations have been shown empirically to improve transfer, adversarial resistance, and generalization out-of-distribution.
• Interpretability: Operating in the frequency domain clarifies the role of visual or structural texture in both natural and adversarial data.

Continued research is expected to focus on learnable frequency bases, self-adaptive injection policies, and integration of frequency-aware paradigms with meta-learning, lifelong learning, and neural architecture search.