Low-Frequency Weighting of Loss Functions

• Low-frequency weighting of loss functions is a method that adjusts loss contributions across frequency bands based on empirical data or human perceptual criteria.
• It employs mathematical formulations like frequency-aware cross entropy and spectral-weighted losses to enhance performance in NLP, speech, and image tasks.
• This approach impacts gradient distribution and optimization dynamics, leading to improved diversity, perceptual quality, and convergence rates in various applications.

Low-frequency weighting of loss functions refers to the systematic application of frequency-dependent scaling within the objective functions employed in machine learning and signal processing. Rather than treating all frequencies equally, loss contributions associated with particular frequency bands—typically, low frequencies—are either up-weighted or down-weighted to realize a desired inductive bias. This principle is fundamental across domains such as natural language processing, speech enhancement, image reconstruction, and numerical linear algebra, with the specific weighting scheme determined by modeling goals (e.g., diversity, perceptual quality, conditioning).

1. Mathematical Formulations and Core Schemes

Low-frequency weighting generally manifests through explicit multiplicative factors within standard loss definitions. Representative mechanisms include:

• Token-frequency weighting in cross-entropy loss: In the context of discrete sequence modeling, cross-entropy (CE) may be generalized to the frequency-aware cross-entropy (FACE), where each token $c_i$ receives a weight $w_i$, typically set inversely proportional to its empirical or output frequency (Jiang et al., 2019):

$$\mathrm{FACE}(y_t) = -\sum_{i=1}^N w_i\, \delta_i(y_t) \log P(c_i \mid y_{<t}, X)$$

with $w_i < 1$ for high-frequency (common) tokens and $w_i > 1$ for rare tokens.

• Frequency-weighted SDR for speech: The weighted SDR loss for time-frequency (TF) representations modifies uniform averaging over the TF grid by applying a positive weight map $w(f, t)$ (Monir et al., 23 Jun 2025):

$$L_{fwsdr}(\hat s) = -10 \log_{10} \left( \frac{\sum_{f, t} w(f, t) \, |S_{proj}(f, t)|^2}{\sum_{f, t} w(f, t) \, |E_{dist}(f, t)|^2} \right)$$

$w(f, t)$ is chosen to stress particular frequencies, such as $w_{low}(f) \propto 1/f^{\alpha}$ over a band, or according to ANSI band-importance weights.

• Spectral-weighted Frobenius objectives: For matrix approximation, the loss

$L_A(X) = \| (X-A)A^{-1} \|_F^2$

penalizes error directions $w_i$0 according to $w_i$1, i.e., up-weights modes associated with small eigenvalues (low spatial frequencies in PDEs) (Trifonov et al., 20 Sep 2025).

• Sobolev-space loss and Focal Frequency Loss: In continuous or grid-based domains, generalized Sobolev norms $w_i$2 or per-frequency weighting with functional forms (e.g., polynomial/radial boosting of low frequencies) are used to accentuate or suppress particular frequency content (Yu et al., 2022, Jiang et al., 2020).

2. Weight Construction and Design Principles

Low-frequency weighting requires the careful construction of frequency-dependent weight maps $w_i$3:

• Empirical or model-based frequency estimation: Token frequencies may be obtained from ground-truth data or model outputs. Output-based counts enable penalizing model-specific frequency usage, critical for diversity-enhancing objectives (Jiang et al., 2019).
• Human perceptual criteria: Loudness contours (e.g., 40-phon curve) motivate weights for speech/audio that counteract the ear's insensitivity to low frequencies, thereby preventing overfitting to inaudible low-frequency errors (Li et al., 8 Nov 2025).
• Spectral/analytic weighting: Polynomial decay, power laws $w_i$4, and anisotropic band scaling are used to systematically emphasize low-frequency bins in TF or DFT domains (Monir et al., 23 Jun 2025, Jiang et al., 2020).
• Adaptive schemes: Per-batch or per-epoch weight maps can be constructed based on current signal/noise ratios or local residual errors, enabling context-sensitive frequency emphasis (Monir et al., 23 Jun 2025).

Representative table summarizing weight construction:

Domain	Weight Source/Function	Frequency Emphasis
NLP (FACE)	Output frequency, Pre-weight	Up-weight rare tokens
Speech Enhancement	Power-law $w_i$5, ANSI	Boost low TF bins
Image Synthesis	Radial, Band-specific in DFT	Boost global/structure
Linear Algebra	$w_i$6 in A eigenbasis	Penalize low-λ/low-freq

3. Theoretical and Algorithmic Properties

Low-frequency weighting schemes modify both the optimization landscape and the dynamics of learning or approximation:

• Gradient rebalancing in sequence modeling: In standard CE, frequent tokens dominate the loss due to their prevalence, biasing the model toward generic, high-frequency outputs and low-diversity responses. By up-weighting rare tokens, as in FACE, the loss gradient is redistributed to underrepresented lexical choices, directly driving output diversity (Jiang et al., 2019).
• Spectral filtering in iterative solvers: Weighted Frobenius objectives with $w_i$7 reweighting act as spectral filters, driving the approximation error out of low-λ modes (low frequencies) and concentrating it in the least-penalized, high-frequency modes. This delivers sparser, better-conditioned preconditioners for CG solvers (Trifonov et al., 20 Sep 2025).
• Perceptual and denoising goals: Low-frequency emphasis or de-emphasis, in losses such as Loud-loss or Focal Frequency Loss, aligns model fit with perceptual or application-specific importance: accentuating mid or high-frequency bands in audio for audibility (Li et al., 8 Nov 2025), or ensuring global structural fidelity in images (Jiang et al., 2020).

4. Implementation Strategies and Recipe Summaries

Practical use of low-frequency weighting is domain-dependent, but common steps include:

1. Obtain/define frequency bins: STFT for TF analysis (audio), DFT for image domains, spherical harmonics for manifold data (Monir et al., 23 Jun 2025, Jiang et al., 2020, Yu et al., 2022).
2. Estimate/statistically derive weights: e.g., empirical counts for tokens, equal-loudness or ANSI tables for audio, polynomial/radial masks for images.
3. Integrate weights into loss: Apply as a per-coefficient or per-segment multiplier in the loss function (see pseudocode samples for PyTorch-style implementations in (Monir et al., 23 Jun 2025, Jiang et al., 2020, Li et al., 8 Nov 2025)).
4. Normalization: Normalize weights (sum to unity or mean 1) to avoid altering the scale of gradients and facilitate stable training.

For example, in Focal Frequency Loss, the focal weight $w_i$8 can be further multiplied by a radial prior or band-specific factor to emphasize low frequencies (Jiang et al., 2020).

5. Empirical Evidence and Performance Impact

Low-frequency-weighted losses empirically influence both quantitative and qualitative outcomes:

• Dialogue generation models using FACE yield substantial increases in diversity metrics (distinct-1: 2.70 → 4.32%, distinct-2: 8.63 → 20.47% on OSDb), while preserving or improving BLEU and human preference vs. strong baselines (Jiang et al., 2019).
• Speech enhancement with frequency-weighted SDR loss produces up to +2–3 dB gains in phoneme-level SDR for plosives/fricatives, and ANSI weighting improves STOI from 0.64 to 0.77 (Monir et al., 23 Jun 2025).
• Loud-loss for audio leads to significant gains in perceptual quality (WB-PESQ: 2.17 → 2.93, ESTOI: 0.812 → 0.836), exceeding uniform or PCS-derived band-importance weighting (Li et al., 8 Nov 2025).
• Spectral-weighted Frobenius loss accelerates PCG convergence and eliminates small eigenvalues in $w_i$9, matching theoretical predictions for low-frequency suppression (Trifonov et al., 20 Sep 2025).
• Focal Frequency Loss in vision improves PSNR/SSIM, FID, and LPIPS in autoencoding, VAE, and GAN benchmarks, particularly recovering low-frequency structure and difficult high-frequency content (Jiang et al., 2020).
• Sobolev-weighted losses in NTK theory provide tunable frequency bias, enabling selective suppression of high-frequency noise ($$\mathrm{FACE}(y_t) = -\sum_{i=1}^N w_i\, \delta_i(y_t) \log P(c_i \mid y_{<t}, X)$$0), uniform learning ($$\mathrm{FACE}(y_t) = -\sum_{i=1}^N w_i\, \delta_i(y_t) \log P(c_i \mid y_{<t}, X)$$1), or high-frequency acceleration ($$\mathrm{FACE}(y_t) = -\sum_{i=1}^N w_i\, \delta_i(y_t) \log P(c_i \mid y_{<t}, X)$$2) (Yu et al., 2022).

6. Applications and Domain-Specific Considerations

Low-frequency weighting is widely applied in:

• Natural Language Generation: Promoting lexical diversity and mitigating generic responses via frequency-aware token weighting (Jiang et al., 2019).
• Speech and Audio Processing: Targeting perceptual quality and intelligibility, e.g., via ANSI, equal-loudness, or power-law frequency scales (Monir et al., 23 Jun 2025, Li et al., 8 Nov 2025).
• Image Synthesis and Reconstruction: Ensuring accurate global structure or texture recovery using DFT-based focal or radial-weighted losses (Jiang et al., 2020).
• Iterative Linear Solvers: Designing preconditioners that specifically optimize low-frequency error attenuation (Trifonov et al., 20 Sep 2025).
• Controlled Bias in Neural Network Training: Adjusting learning dynamics toward or away from low-frequency representations in overparameterized networks, particularly in nonuniform data regimes (Yu et al., 2022).

7. Limitations and Practical Challenges

The efficacy of low-frequency weighting depends on:

• Correctly modeling domain-specific importance: Perceptual weighting must use accurate psychoacoustic or visual models (e.g., true equal-loudness contours), as weaker proxies yield diminished gains (Li et al., 8 Nov 2025).
• Potential trade-offs: Overweighting one end of the spectrum can degrade performance elsewhere. For instance, boosting high-frequency fit for deblurring may amplify noise unless regularized (Yu et al., 2022).
• Normalization and stability: Improper normalization or overly sharp weighting may destabilize training or lead to mode collapse, especially in deep architectures (Jiang et al., 2020).
• Computational cost: Frequency-domain transformations (e.g., full 2D DFTs) introduce minor but non-negligible overheads, and batch- or epoch-wise adaptation requires additional memory or computation (Jiang et al., 2020, Monir et al., 23 Jun 2025).

Overall, low-frequency weighting of loss functions constitutes a versatile and theoretically well-grounded strategy for tailoring model fit to domain-specific goals, influencing both learning dynamics and final model behavior across a diverse range of applications (Jiang et al., 2019, Monir et al., 23 Jun 2025, Trifonov et al., 20 Sep 2025, Li et al., 8 Nov 2025, Yu et al., 2022, Jiang et al., 2020).