Frequency-Domain MLPs

• Frequency-domain MLPs are neural network architectures that operate in the frequency space by leveraging sinusoidal encodings and spectral transformations to overcome the spectral bias of standard MLPs.
• They incorporate methods like input frequency encoding, periodic activations with tunable frequencies, and direct operation in transformed domains to enhance the modeling of rapidly oscillatory signals.
• Empirical evaluations show improved high-frequency reconstruction, better sample efficiency, and reduced error in applications such as time series forecasting and implicit neural representations.

A frequency-domain Multi-layer Perceptron (MLP) is any neural network architecture or training methodology in which major components, operations, or representations operate in the frequency domain—typically via explicit transformation, specialized activations, or frequency-driven architectural constraints. Frequency-domain MLPs appear in a variety of application contexts (e.g., implicit neural representations, time series forecasting, scientific computing), exploit signal processing principles (Fourier, Cosine, Sine expansions, spectral sampling), and address longstanding limitations of conventional MLPs regarding high-frequency expressivity, generalization, and sample efficiency.

1. Theoretical Motivations and Spectral Bias

Standard coordinate-based MLPs exhibit a well-documented “spectral bias”—a pronounced preference for learning low-frequency (smooth) components of functions before higher-frequency (rapidly oscillatory) ones. This is a consequence of the structure of the neural tangent kernel (NTK) for typical architectures: eigenmodes corresponding to high frequencies have smaller eigenvalues and thus converge slowly during gradient-based training. Consequently, traditional architectures are insufficient for applications requiring high-fidelity modeling of rapidly varying signals (Tancik et al., 2020).

To mitigate spectral bias, frequency-domain MLPs adopt strategies such as input feature mapping using sinusoids (Fourier features or positional encodings), periodic activations with explicit frequency controls (e.g., SIREN, FINER), or direct operation in the transformed (Fourier, Cosine) representation of the data. These approaches are unified by their explicit handling of frequency content, either by enriching the representational basis or by directly parameterizing the underlying frequency spectrum.

2. Core Architectural Variants

Frequency-domain MLPs can be broadly grouped according to how frequency considerations are incorporated:

a. Input Frequency Encoding:

Injecting frequency information at the input stage (e.g., Fourier features, positional encoding) lifts the input into a higher-dimensional space constructed from sinusoidal basis functions with chosen frequencies. A typical positional encoding maps input $x\in\mathbb{R}^d$ to

$$\text{PE}(x) = [\sin(2^0\pi x),\, \cos(2^0\pi x),\, \ldots,\, \sin(2^{M-1}\pi x),\, \cos(2^{M-1}\pi x)]$$

This mapping makes higher frequencies directly accessible to the MLP, thus increasing its ability to represent non-smooth features (Lin et al., 2024, Tancik et al., 2020).

b. Periodic Activations with Frequency Multipliers:

Periodically activated MLPs use sine (or cosine) nonlinearities with tunable frequencies:

$\sigma(x) = \sin(\omega x)$

where $\omega$ is often a layer-wise or neuron-specific hyperparameter (Alsakabi et al., 27 Sep 2025). Recent variants such as FM-SIREN and FM-FINER introduce neuron-specific, Nyquist-informed multipliers to span the frequency basis efficiently, reducing hidden feature redundancy by ≈50% and improving expressive efficiency in 1D/2D/3D INR tasks (Alsakabi et al., 27 Sep 2025). In contrast, classical SIREN uses a fixed $\omega$, limiting bandwidth and causing redundancy.

c. Frequency-Domain Transformation and Learning:

Transformation of the input data (or hidden states) to the frequency domain via the (D)FFT or DCT, followed by learning on the spectral coefficients, yields architectures such as FreTS (for time series, operating on real and imaginary parts of the complex spectrum) and FSMLP (Discrete Cosine Transform, DCT-II, with simplex-constrained channelwise layers) (Yi et al., 2023, Li et al., 2024). This exploits properties such as global temporal and inter-channel receptive fields, energy compaction, and, via the convolution theorem, efficient modeling of periodic dependencies.

d. Frequency-Adaptive or Bandlimited Architectures:

Architectures that learn or explicitly control the set of frequencies used by the network, encompassing nonharmonic bases (Fourier Learning Machines, FLMs), trainable spectral sampling, and block-recursive frequency synthesis (Fibonacci Networks) (Rubel et al., 10 Sep 2025, Bleiberg et al., 2024). FLMs, for example, parameterize each neuron/unit with learnable frequencies, amplitudes, and phase shifts, constructing a nonharmonic Fourier series adapted to the data.

3. Frequency-Domain Analysis and Sampling Principles

Frequency-domain MLPs enable principled analysis and control of the spectral properties of learned functions. The key insight is the existence of an intrinsic bandwidth or cutoff frequency for a given architecture, determined by the encoding, the linear transformations (product of layer norms), and the activation nonlinearity.

Intrinsic Cutoff and Sampling Rate:

For PE-MLPs, the intrinsic frequency $\omega_{\text{intrinsic}}$ can be estimated via:

• Randomly initializing the network, sampling a dense grid, computing the output, and applying a DFT to estimate $\lvert F(\omega) \rvert$.
• Identifying the maximum frequency for which the spectrum is above a fixed fraction of its peak.
• The Nyquist–Shannon theorem then imposes the necessary sampling interval $\Delta x \leq \pi / \omega_{\text{intrinsic}}$ (Lin et al., 2024).

Experiments confirm that, for SDF fitting, reconstruction error plateaus once the optimal sampling determined by this procedure is reached, and further densification does not reduce fitting error (Lin et al., 2024).

4. Representative Architectures and Empirical Results

Architecture / Principle	Frequency Operation	Distinctive Features
Positional Encoding MLP	Input Fourier features	Tunable input bandwidth; prone to artifacts (Tancik et al., 2020, Lin et al., 2024)
FM-SIREN/FM-FINER	Neuron-specific periodic activations	Nyquist-informed coverage; 50% less redundancy (Alsakabi et al., 27 Sep 2025)
FreTS	DFT/Freq-domain MLP layers	Complex-valued MLP; multi-scale learning (Yi et al., 2023)
FSMLP	DCT-frequency domain; simplex MLP	Simplex-constrained channel MLP; reduced Rademacher complexity (Li et al., 2024)
FLM (Fourier Learning Machine)	Learnable cosine basis	Adaptive nonharmonic spectrum; direct Fourier-param mapping (Rubel et al., 10 Sep 2025)
Fibonacci Network	Block-recursive frequency synthesis	Exploits trigonometric identities; no PE (Bleiberg et al., 2024)

Extensive benchmarks demonstrate that frequency-domain MLPs often outperform recurrent, convolutional, and attention-based architectures in applications such as

• Time series forecasting: FreTS achieves up to 9.4% lower MAE and 11.6% lower RMSE than next-best models on standard benchmarks (Yi et al., 2023); FSMLP reports significant error reduction and scalability (Li et al., 2024).
• Implicit neural representations (images, audio, SDFs, NeRFs): FM-SIREN/FM-FINER improve PSNR and IoU over both fixed-ω SIREN and standard PE methods (Alsakabi et al., 27 Sep 2025).
• Scientific machine learning (PDEs, OCPs): FLMs converge faster and to lower error than SIREN or vanilla MLPs, due to their problem-adapted nonharmonic basis (Rubel et al., 10 Sep 2025).
• Robust high-frequency reconstruction: Fibonacci Networks avoid high-frequency artifacts and demonstrate superior noise robustness compared to PE-equipped MLPs (Bleiberg et al., 2024).

5. Frequency Control, Capacity Bounds, and Generalization

Frequency-domain MLPs provide explicit handles to control expressivity and generalization:

• Spectral capacity in periodic MLPs is analyzable via amplitude-phase expansions; multilayer compositions produce harmonics as integer combinations of base frequencies, controlled by weight magnitudes via Bessel function amplitude bounds (Novello et al., 2024).
• Hard and soft clamping (e.g., TUNER training for sinusoidal networks) restrict hidden-layer weight magnitudes to attenuate undesired high harmonics, preventing overfitting and ensuring stable convergence (Novello et al., 2024).
• Imposing convex constraints (e.g., simplex constraints in FSMLP) provably reduces the Rademacher complexity of the linear maps, tightening generalization (Li et al., 2024).
• Adaptive spectral initialization and learning: FLMs and TUNER select spectral basis elements and associated bounds based on anticipated task bandwidth, further aligning learning capacity with task requirements (Rubel et al., 10 Sep 2025, Novello et al., 2024).

6. Trade-offs and Design Considerations

• Memory and Model Size:

Frequency lifting (positional encoding) increases input dimension, while grid-based or local positional encodings strike a balance between frequency range, locality, and memory (Fujieda et al., 2023).

• Spectral Artifacts:

Overly high frequency encodings without appropriate sampling or regularization can induce ringing or noisy artifacts (Gibbs phenomenon), highlighting the necessity of applying bandlimiting and frequency-aware sampling (Lin et al., 2024, Novello et al., 2024).

• Redundancy vs. Coverage:

Fixed-frequency approaches (SIREN) induce feature redundancy; neuron-specific frequency allocation (FM-SIREN, FM-FINER), block recurrence (Fibonacci Networks), or grid modulation (Local Positional Encoding) break this redundancy and achieve more efficient spectral coverage (Alsakabi et al., 27 Sep 2025, Bleiberg et al., 2024, Fujieda et al., 2023).

• Application-Specific Transform Choices:

Time series forecasting often prefers real-valued transforms (DCT), while implicit field or scientific ML problems typically exploit Fourier or sine/cosine expansions based on the signal structure (Yi et al., 2023, Li et al., 2024, Rubel et al., 10 Sep 2025).

7. Outlook, Open Problems, and Emerging Directions

• Generalization Theory:

The theoretical understanding of sample complexity and generalization for frequency-domain MLPs—particularly with learned or nonharmonic spectral bases—remains nascent (Yi et al., 2023).

• Irregular Sampling and Nonuniform Data:

Most current frameworks assume uniform sampling; generalization to nonuniform, irregular, or graph-based inputs requires new spectral methods and adaptivity (Yi et al., 2023).

• Transfer to Other Modalities:

Frequency-domain MLPs show promise for anomaly detection, imputation, classification in time series, and extensions to spatio-temporal signals and point clouds (Yi et al., 2023, Li et al., 2024).

• Learned Hybrid Representations:

New architectures may further combine learned bases with classical ones, or integrate spectral methods with graph/geometry-aware neural networks for high-dimensional data (Alsakabi et al., 27 Sep 2025, Rubel et al., 10 Sep 2025).

Frequency-domain MLPs represent a mature and rapidly evolving paradigm, systematically leveraging spectral representations, frequency analysis, and signal processing principles to substantially enhance the expressive power, efficiency, and robustness of neural function approximators across a range of domains (Lin et al., 2024, Yi et al., 2023, Alsakabi et al., 27 Sep 2025, Li et al., 2024, Tancik et al., 2020, Novello et al., 2024, Fujieda et al., 2023, Rubel et al., 10 Sep 2025, Bleiberg et al., 2024).