Fourier Preconditioning for Neural Feature Learning

Abstract: Mutual information (MI)-inspired feature learning techniques are capable of generating low-dimensional embeddings that retain nonlinear dependence structures, but direct estimations of MI suffer from noisy probability distribution estimates in the low-data regime. The H-Score objective, computed from second-order statistics, provides a practical proxy metric for training feature extraction networks. We prove that H-Score is invariant to invertible transformations in the unrestricted functional setting, but becomes sensitive to input basis rotations under constrained approximation classes. Consequently, we study unitary preconditioning for H-Score networks and show that selecting an appropriate basis rotation reduces finite-width truncation error by concentrating predictive dependence into fewer dominant modes. We identify the fast Fourier transform (FFT) as an effective data-independent, low-cost preconditioner for approximately stationary processes, where spectral structure induces concentration of the cross-covariance singular value spectrum. We introduce training-free metrics based on spectral entropy and cumulative dependence energy to quantify basis suitability and predict downstream inference gains prior to network training. Experiments across eight multivariate datasets demonstrate that FFT preconditioning is particularly useful in resource-constrained regimes, achieving up to 50% normalized mean squared error (NMSE) reduction, while the proposed metrics correlate with observed performance gains and correctly identify cases where spectral preconditioning is detrimental.

• The paper introduces spectral preconditioning using FFT to minimize truncation error in neural feature learning.
• It leverages entropy-based metrics like the entropy ratio and cumulative energy rank to predict when FFT preconditioning is beneficial.
• Empirical results show up to a 50% reduction in NMSE in low-data conditions, outperforming randomized basis rotations.

Fourier Preconditioning for Neural Feature Learning: A Technical Synthesis

Motivation and Context

Resource constraints in feature learning motivate the development of mechanisms that enable neural networks to extract informative, low-dimensional embeddings while preserving nonlinear statistical dependencies. Although mutual information (MI)-maximizing approaches provide a theoretically robust framework for generating such representations, direct MI estimation is unreliable in low-data regimes due to estimation noise. The H-Score objective, based on second-order statistics, has emerged as a practical surrogate. This work rigorously studies the invariance properties of the H-Score objective to invertible transformations in theory, and empirically demonstrates how basis choice strongly impacts practical finite-width neural approximations. The central focus is the introduction and analysis of spectral preconditioning—specifically, fast Fourier transform (FFT)-based rotation—as an effective, data-independent basis choice that minimizes truncation error for approximately stationary processes (2607.02199).

Theoretical Foundations

H-Score Objective and Basis Invariance

The H-Score objective is founded on the Hilbert-Schmidt (HS) dependence energy extracted from the canonical dependence kernel (CDK) linking two random variables. It serves as a lower-rank approximation to MI by maximizing orthogonal dependence modes that are most informative for the downstream prediction task. When considered in function space and with unrestricted function classes, invariance to invertible basis transformations is proven: the supremum of the H-Score remains unchanged under such transformations. However, when neural network realizations impose restrictions—such as finite width, nonlinearity, and parameterization—the invariance breaks down. Truncation errors in the rank-$k$ approximation become sensitive to basis alignment with the dominant singular modes of the cross-covariance operator.

Spectral Preconditioning and the FFT

The paper demonstrates that for data with approximate stationarity and spectral concentration, a unitary transformation such as the FFT approximately diagonalizes the cross-covariance matrix. Under these conditions, dominant dependence energy is concentrated into a small set of spectral modes. This property ensures that truncation to the top $k$ modes preserves much of the relevant information, directly reducing finite-width approximation error. In contrast, approaches such as PCA or CCA require empirical covariance estimates that are unreliable with limited data and are further constrained to capturing only linear dependencies.

Methodology

Metrics for Basis Suitability

To determine whether spectral preconditioning will be beneficial for a given dataset, the study introduces training-free metrics evaluable before model training:

• Spectral Entropy: Quantifies concentration of predictive information based on the singular value spectrum of the cross-covariance matrix in both temporal and frequency domains.
• Entropy Ratio (ER): The ratio of frequency to temporal domain spectral entropy; values less than one indicate greater concentration—and thus greater likely benefit—of spectral preconditioning.
• Cumulative Energy Rank (CER): The smallest latent dimensionality needed to retain a prescribed percentage of dependence energy, informing the optimal complexity of downstream models.

Empirical Analysis

Experiments are conducted on eight multivariate datasets from domains including solar power, traffic, climate, wireless networking, and tabular regression. Temporal inputs are segmented, transformed via FFT where applicable, and mapped to forecast targets. Feature extraction is performed by jointly trained Echo State Networks (ESN) and fully connected MLPs, explicitly examining the effect of preconditioning on learned feature performance under resource-constrained (low-data) and high-data regimes.

Results

Numerical Performance Metrics

The results demonstrate that FFT preconditioning yields strong numerical improvements—up to a 50% reduction in normalized mean squared error (NMSE) on specific datasets—primarily in severe data-limited conditions. Datasets with stationary or spectrally localized dependence benefit the most, confirmed objectively by the proposed entropy ratio and CER metrics. In contrast, non-stationary or aperiodic data (e.g., wine quality datasets) sometimes experience degraded performance from FFT preconditioning, an outcome predicted a priori by high entropy ratios.

Significantly, randomized basis rotations fail to match the gains of FFT preconditioning, supporting the claim that specific basis selection informed by data structure, rather than arbitrary rotation, is critical for improved finite-width network performance.

Implications for Model Design

The empirical findings support several actionable conclusions:

• Low-data regimes: FFT preconditioning consistently and strongly improves H-Score network performance where data scarcity impairs covariance estimation.
• Spectral metrics as predictors: The entropy ratio and CER effectively predict both the suitability and likely magnitude of spectral preconditioning benefits, guiding practitioners on when to employ FFT.
• Architectural considerations: The ESN, when paired with FFT inputs, captures inter-frequency coupling relations, enabling richer feature representations despite dimension constraints.

Theoretical and Practical Implications

Theoretical Insights

The formal proof of theoretical invariance of H-Score to invertible basis transformations, alongside practical demonstration of basis dependence in restricted function classes, provides a principled explanation for longstanding empirical observations in neural feature learning. It bridges the gap between classical statistical signal processing (which leverages spectral localization) and modern neural representation learning, introducing a pathway for hybrid methodologies.

Practical Consequences and Prospective Developments

FFT-based preconditioning offers a computationally efficient, universal, and data-agnostic approach for improving neural feature extraction in stationary temporal domains, circumventing the unreliability of empirical covariance-based procedures in small-sample settings. The proxy metrics for basis evaluation lay a foundation for automated, data-driven basis selection in pipeline design.

Open directions include:

• Developing methods for optimal latent dimension prediction informed by spectral metrics.
• Extending spectral preconditioning to alternative architectures beyond ESN-MLP compositions.
• Investigating windowing strategies to mitigate spectral leakage for short-time FFTs.
• Studying preconditioning for non-stationary or multi-scale temporal processes using generalized spectral transforms.

Conclusion

This study rigorously analyzes the utility of Fourier preconditioning in neural feature learning with the H-Score objective, demonstrating that spectral preconditioning minimizes finite-width truncation error when the dependence structure is stationary or spectrally localized. Theoretical results clarify basis invariance under idealized circumstances and provide the justification for targeted basis alignment in practical settings. The entropy-based proxy metrics allow for a priori prediction of preconditioning effectiveness, supporting automated, metric-driven basis choice. The strong empirical gains in low-data regimes highlight the relevance of FFT preconditioning in resource-constrained inference scenarios. This work formalizes the interface between neural representation learning and classical signal processing, providing foundations and tools for improved feature learning pipelines (2607.02199).