Fourier Feature Embedding (FFE) Trunk

• Fourier Feature Embedding (FFE) trunk is a neural component that projects input data into a high-dimensional sinusoidal space, enabling efficient capture of high-frequency and oscillatory structures.
• It includes both fixed random and trainable variants, each balancing computational efficiency with enhanced spectral expressivity and improved NTK conditioning.
• FFE trunks stabilize gradient scales and accelerate convergence across domains by pre-aligning network predictions, benefiting applications from tabular learning to scientific simulations.

A Fourier Feature Embedding (FFE) trunk is a neural architecture component that projects input coordinates or tabular data into a high-dimensional space of sinusoidal features, enabling neural networks—particularly MLPs—to efficiently capture high-frequency and oscillatory structure. FFE trunks play a key role in mitigating spectral bias, accelerating convergence, conditioning the neural tangent kernel (NTK), and, depending on their formulation, may introduce fixed or trainable sinusoidal bases. Their design is central to advances in representation learning across tabular data, physical systems (PINNs, operator learning), implicit neural representations, and reinforcement learning.

1. Mathematical Formulation of the FFE Trunk

Let $x \in \mathbb{R}^d$ be an input. The canonical FFE trunk applies a fixed, parameter-free mapping: $$\Phi(x) = \sqrt{\frac{2}{D}} \begin{bmatrix} \sin(\Omega^\top x) \ \cos(\Omega^\top x) \end{bmatrix} \in \mathbb{R}^{2D}$$ where $$\Omega = [\omega_1,\ldots,\omega_D] \in \mathbb{R}^{d \times D}$$ and $\omega_i \sim p(\omega)$, the Fourier-transform of a chosen shift-invariant kernel $k(\cdot)$ (commonly the Gaussian for RBF kernels: $\omega_i \sim \mathcal{N}(0, \sigma^{-2} I_d)$) (Sergazinov et al., 3 Jun 2025).

Alternative variants include learnable frequency matrices $B$ or per-coordinate frequencies $\{f_i\}$ (Tang et al., 2024), as well as axis-aligned, multi-resolution positional encodings for coordinate-based applications. The FFE trunk may admit slight variations, such as bias terms in phase or concatenation with the original input (to avoid information loss) (Li et al., 2021).

Fixed FFE architectures use random, non-trainable $\Omega$, providing a plug-and-play, parameter-free mapping. In contrast, learned FFE trunks introduce $\Omega$ (or its low-dimensional analogues) as parameters, with gradients computed via backpropagation (Tang et al., 2024, Li et al., 2021). The output of the FFE trunk replaces or augments the raw input to a downstream neural network, typically an MLP or transformer.

2. Theoretical Mechanisms and Neural Tangent Kernel Conditioning

Standard multilayer perceptrons (MLPs) exhibit spectral bias: under both finite and infinite-width NTK analysis, low-frequency (smooth) components in the target function converge rapidly, while high-frequency modes are severely attenuated (Tancik et al., 2020, Sergazinov et al., 3 Jun 2025). The NTK for a typical MLP lacks translation invariance, exhibits unboundedness with respect to input norm, and is poorly conditioned for high frequencies.

The FFE trunk fundamentally alters these properties. Transformed inputs $\Phi(x)$ induce an NTK

$$\tilde K_t(x, x') = \nabla_\theta f_{\theta_{t-1}}(\Phi(x))^\top \nabla_\theta f_{\theta_{t-1}}(\Phi(x'))$$

which, by construction, is shift-invariant and stationary, with a spectrum closely aligned to the sampled frequencies in $\Omega$ (Tancik et al., 2020). In the fixed FFE case, the empirical NTK spectrum is both bounded and well conditioned: Theorem 1(2) in (Sergazinov et al., 3 Jun 2025) establishes

$\sup_{x,x'} |\tilde K_t(x, x')| < \infty$

This bounded spectrum prevents the pathology of exploding gradients and aligns the network's functional prior to the spectral decomposition induced by the chosen kernel.

The telescoping expansion of network output over training steps reveals that the FFE trunk's fixed kernel component ($m_t$) pre-aligns the initial predictions with the target, introducing a beneficial bias and shortening the optimization path. In high-frequency regression tasks and PDE surrogates, this explicitly enables accurate and rapid learning of oscillatory components that would otherwise converge slowly (or not at all).

3. Fixed vs. Trainable FFE Trunks

FFE trunks can be classified as follows:

Trunk Class	Frequency Parameters	Embedding Update
Fixed Random (RFF)	$\omega$ sampled i.i.d.	Not updated
Axis-aligned/Hand-tuned (PE)	Fixed grid (often powers)	Not updated
Learned/Trainable (LFF)	$\omega$ or $f_i$ learned	Joint w/ network

In fixed RFF/PE, the mapping is parameter-free after initialization, computationally efficient, and applicable as a block in any architecture (Sergazinov et al., 3 Jun 2025, Tancik et al., 2020). Trainable FFE trunks, as in PINNs for high-frequency surface lubrication (Tang et al., 2024) or learned Fourier features for RL (Li et al., 2021), optimize the frequencies to better match the spectral content of the data, overcoming the limitations of randomly chosen features especially for complex, high-dimensional or multi-scale problems. Ablation results demonstrate that trainable FFE trunks offer orders-of-magnitude error improvements and robust, rapid convergence on rough or non-smooth targets (e.g., 3.33% vs 81.11% error in textured lubrication (Tang et al., 2024)).

Practical implementations favor D in the 256–1024 range (tabular: (Sergazinov et al., 3 Jun 2025)), and frequency bandwidth tuning (grid search on σ or initialization range) as the main hyperparameter (Tancik et al., 2020, Li et al., 2021, Sojitra et al., 15 Sep 2025).

4. Algorithmic Integration and Training Paradigms

The FFE trunk functions as a near-universal input module across architectures:

1. Sample or initialize frequency parameters (fixed or trainable).
2. Compute $\Phi(x)$ or $\gamma(x)$ via sin/cos basis projection—optionally concatenate with raw input or other features.
3. Pass the resulting vector to the downstream network, without altering its internal weights or connections.
4. In the trainable case, jointly update the trunk’s frequency parameters throughout optimization; in the fixed case, these remain static (Sergazinov et al., 3 Jun 2025, Tang et al., 2024, Sojitra et al., 15 Sep 2025).

The FFE trunk’s output is bounded ($\|\Phi(x)\| \leq \sqrt{2}$), which stabilizes the scale of gradients and enables aggressive learning rate schedules (2–5× higher than with raw input) without instability (Sergazinov et al., 3 Jun 2025).

In operator learning, the trunk replaces coordinate input layers (DeepONet trunk), yielding spectral expressivity at the cost of one extra matrix multiply and sin/cos operations—no additional learnable parameters (Sojitra et al., 15 Sep 2025). In PINNs and RL, it enables learning functions with sharper transitions, faster bootstrapping, and improved sample efficiency (Tang et al., 2024, Li et al., 2021).

5. Empirical Performance Across Domains

FFE trunks deliver substantial gains across data modalities:

• Tabular deep learning: +0.8 to +1.7% absolute accuracy (classification), 10–14% RMSE reduction (regression), accelerated convergence, and reduced need for hyperparameter tuning. All main architectures (MLP, ModernNCA, TabTransformer, SAINT) benefit (Sergazinov et al., 3 Jun 2025).
• Scientific ML (PDEs, PINNs, Operator Learning):
  • DeepONet (FEDONet) achieves 2–3× lower relative L2 error than vanilla trunks on Poisson, Allen-Cahn, Burgers’, Kuramoto-Sivashinsky, Lorenz-96, Eikonal, etc. (Sojitra et al., 15 Sep 2025).
  • PINNs for rough-surface lubrication: trainable FFE trunks reduce maximum error from >80% (fixed) to <4% (trainable) (Tang et al., 2024).
  • High-frequency tomography, MRI, NeRF, and implicit 3D reconstruction tasks demonstrate PSNR/IoU/SSIM improvements of 3–12 dB over baseline MLPs (Tancik et al., 2020, Audia et al., 18 Apr 2025).
• Reinforcement Learning: LFF trunks result in higher sample efficiency, more stable Q-learning in noisy scenarios, controlled spectral regularization, and rapid fitting of high/low-frequency functions (Li et al., 2021).

FFE trunks outperform standard MLPs and fixed grid/traditional positional encodings in high-frequency or rough settings, although for the finest-scale detail, learnable grid-based alternatives such as Multigrid Parametric Encodings (MPE) may further boost the kernel spectrum and signal resolution (Audia et al., 18 Apr 2025).

6. Robustness, Regularization, and Design Extensions

Despite their strengths, FFE trunks can propagate noise when data contain high-frequency measurement error or when the embedding introduces frequency components not present in the target function (Jeong et al., 2024, Ma et al., 8 Feb 2025). Robustification strategies include:

• Diagonal head layers after FFE, yielding implicit regularization and frequency sparsity—this delivers L2 error reductions (0.03–0.04 vs 0.28–0.32 for dense FFE+dense heads) and superior noise generalization (Jeong et al., 2024).
• Adaptive filtering and bias-free MLPs to suppress frequencies amplified by embedding/MLP interactions (Ma et al., 8 Feb 2025).
• Quantized Fourier Features (QFF)—a hybrid of frequency and binning—extend continuity, enable fast convergence/localization, and reduce the parameter count vs spatial grids, while maintaining multi-resolution adaptation (Lee et al., 2022).
• Parsvel regularization in frequency space (e.g., in PREF), which constrains high-frequency coefficient magnitudes and regularizes the induced smoothness of the output function (Huang et al., 2022).

These advances highlight both the strength and the limitations of FFE trunks—suitability for preconditioned learning, spectral bias mitigation, and NTK conditioning, but sensitivity to noise in both data and the embedding itself.

7. Best Practices, Limitations, and Current Research Trajectory

Key guidelines for deploying FFE trunks include:

• Choose D large enough for NTK Gram-matrix accuracy (<1% entrywise error) (Sergazinov et al., 3 Jun 2025).
• Adapt the frequency distribution $p(\omega)$ to the target problem; for RBF-like smoothness, use $\mathcal{N}(0, \sigma^{-2} I_d)$, and grid search σ when possible (Sergazinov et al., 3 Jun 2025, Tancik et al., 2020).
• For high-dimensional or nonstationary problems, consider learnable frequencies or per-axis (tensor-product) embeddings (Tang et al., 2024, Li et al., 2021).
• In presence of noise, use diagonal layers or spectral regularization and avoid excessive frequency bandwidth (Jeong et al., 2024, Ma et al., 8 Feb 2025).
• In tabular or bounded-input domains, do not perform additional feature normalization; the trunk's scaling ensures bounded output variance (Sergazinov et al., 3 Jun 2025).
• For data with both high- and low-frequency content, use a mixture of frequency scales for initialization, or adopt multiresolution schemes (Tang et al., 2024).

While FFE trunks have become ubiquitous for their simplicity, plug-and-play architecture, and pronounced empirical benefits, research continues into learnable grid-based representations (MPE), higher-order harmonics, domain-specific spectral adaptation, and stable training regimes for extreme frequency or high-noise synthesis (Audia et al., 18 Apr 2025, Lee et al., 2022, Ma et al., 8 Feb 2025). Their interaction with deeper architectures, transformers, and neural operator models remains a rich area of investigation.

---

In summary, the Fourier Feature Embedding trunk is a central architectural tool for inducing spectral expressivity, NTK conditioning, and rapid convergence in neural networks for tabular, scientific, and coordinate-based learning. Its design accommodates a spectrum from fixed, data-agnostic mappings to fully trainable and robustified variants, with demonstrable theoretical and empirical gains across domains (Sergazinov et al., 3 Jun 2025, Tang et al., 2024, Tancik et al., 2020, Sojitra et al., 15 Sep 2025, Jeong et al., 2024).