Fourier Domain HPO Techniques

• Fourier Domain Hyper-Parameter Optimization is a collection of techniques that leverage spectral representations to simplify and accelerate the hyperparameter tuning process in complex numerical and ML models.
• These methods utilize Fourier surrogate modeling, quantum-enhanced regression, and adaptive minimax strategies to reduce sample complexity and achieve significant computational speedups.
• Applications span from optimizing PDE solvers to fine-tuning Fourier Neural Operators, providing robust performance improvements and scalable solutions in high-dimensional settings.

Fourier Domain Hyper-Parameter Optimization is a collection of methodologies that leverage Fourier-analytic, spectral, or frequency-domain representations to facilitate, accelerate, or stabilize hyperparameter optimization in numerical algorithms and machine learning models. These techniques are especially pertinent to applications in numerical PDE solvers (particularly multigrid methods), spectral neural operator architectures (e.g., Fourier Neural Operators, or FNOs), and high-dimensional machine learning systems where traditional search methods become computationally prohibitive.

1. Fourier-Analytic Foundations and Motivation

Hyperparameter optimization (HPO), in both numerical computing and machine learning, often involves searching over a high-dimensional, nonconvex, and expensive-to-evaluate response landscape. The structure of this landscape frequently admits a parsimonious or sparse representation in a suitable Fourier-analytic or spectral basis. For example, in Boolean configuration spaces or continuous parameter domains, the objective function can be expanded as a (tensor-product) Fourier or orthogonal polynomial series, exploiting low-degree or sparse dependence among hyperparameters (Hazan et al., 2017).

In numerical PDE algorithms, such as multigrid or domain decomposition, the underlying method performance (e.g., asymptotic convergence rates or condition numbers) can be explicitly expressed as functions of both algorithmic hyperparameters and frequency-domain variables, enabling direct optimization in the Fourier space (Brown et al., 2020). In operator learning, neural architectures like FNOs parameterize their core mappings in the Fourier domain, making the sensitivity of model behavior to spectral hyperparameters (e.g., number of retained Fourier modes) a primary concern (Li et al., 24 Jun 2025, Sun et al., 2024).

2. Spectral Methods in Hyperparameter Optimization Algorithms

Spectral approaches to HPO can be classified as either direct (explicitly manipulating Fourier/spectral symbols) or indirect (using Fourier features as function surrogates).

2.1. Spectral or Fourier Surrogate Modeling

HPO is recast as a sparse polynomial regression problem in an orthogonal basis adapted to the hyperparameter domain. For Boolean, categorical, or continuous variables, this involves expanding the response function as

$f(x) = \sum_S \hat{f}_S \psi_S(x),$

where $\{\psi_S\}$ is a tensor-product orthogonal basis (e.g., parity functions for $\{-1,1\}^n$, Hermite polynomials for $\mathbb{R}^n$), and $\hat{f}_S$ are the spectral coefficients. Compressed sensing techniques, such as Lasso, are then applied to a small sample of function evaluations to recover the dominant coefficients, yielding a low-dimensional surrogate model for efficient minimization (Hazan et al., 2017). This spectral recovery reduces sample complexity and allows for embarrassingly parallel evaluation, often outperforming Bayesian Optimization and Random Search in high-dimensional spaces with low-effective-degree structure.

2.2. Quantum Fourier and Trigonometric Regression

In quantum-enhanced HPO, variational quantum circuits encode hyperparameters as rotation angles. The observable expectation values define truncated Fourier series in the hyperparameters,

$$f(\mathbf{x}; \theta) = \sum_{\mathbf{k} \in \mathcal{K}_L} A_{\mathbf{k}}(\theta) \cos(\mathbf{k} \cdot \varphi) + B_{\mathbf{k}}(\theta) \sin(\mathbf{k} \cdot \varphi),$$

where the coefficients $A_{\mathbf{k}}, B_{\mathbf{k}}$ are learned classically (Consul-Pacareu et al., 2023). This hybrid quantum-classical protocol first fits the quantum circuit parameters to a small number of sampled evaluations and then locates the optimum of the resulting (analytic) Fourier surrogate. Empirically, this can result in 50–90% wall-time savings versus classical HPO, with comparable generalization performance.

3. Fourier Domain HPO in PDE Solvers: Local Fourier Analysis and Minimax Optimization

For many iterative solvers for PDEs, local Fourier analysis (LFA) provides an explicit frequency-domain expression for the convergence factor as a function of both hyperparameters and Fourier frequencies. The minimax optimization problem,

$$\min_{p \in D} \max_{\theta \in [0,2\pi]^d} \rho(T(p, \theta)),$$

seeks hyperparameters $p$ that minimize the worst-case spectral radius over admissible frequencies. The mapping $T(p, \theta)$ represents the so-called Fourier symbol of the algorithm's error propagator. It follows that efficient hyperparameter optimization requires robust minimax algorithms that can efficiently resolve worst-case frequency angles.

Contemporary methods include:

• Brute-force discretized search: explicit grid evaluation over both $p$ and $\theta$, which scales poorly.
• Fixed-inner minimization: using a reduced grid in frequency, optimizing $p$ by non-smooth optimization (HANSO-FI).
• Adaptive-sampling minimax (ROBOBOA): iteratively constructs the outer approximation by alternately optimizing $p$ for a growing finite set of critical frequencies and adding new “worst-case” $\theta$ (Brown et al., 2020).

These methods efficiently compute optimal relaxation weights, coarse-grid parameters, and other hyperparameters of multigrid and related methods in a manner informed directly by the method’s spectral performance.

4. Fourier Neural Operator-Specific Hyperparameter Optimization

Fourier Neural Operators (FNOs) present unique challenges for HPO due to their explicit parametrization in the Fourier domain. As the number of Fourier modes, $K$, grows—especially in higher-dimensional PDEs—the kernel weight tensor's dimensionality explodes ($K^d \times m \times m$), and the optimal hyperparameters (e.g., learning rate) exhibit instability under conventional scaling.

The Maximal Update Parametrization (μP), and the associated μTransfer-FNO, introduce principled scaling for both weight initialization and optimizer hyperparameters:

$b(K) = c(K) = (d \log K)^{-1/2}$

where $b(K)$ is the initialization standard deviation and $c(K)$ scales the optimizer's “master” learning rate, ensuring that feature updates per layer remain $\Theta(1)$ across all $K$ (Li et al., 24 Jun 2025).

The μTransfer-FNO protocol proceeds by:

1. Hyperparameter search on a small proxy FNO with mode count $K_{\mathrm{proxy}}$.
2. Analytic rescaling of μP-sensitive hyperparameters:

$$\eta^* \leftarrow \eta^*_{\text{proxy}} \times \sqrt{\frac{\log K_{\text{proxy}}}{\log K^*}}$$

$$\sigma^* \leftarrow \sigma^*_{\text{proxy}} \times \sqrt{\frac{\log K_{\text{proxy}}}{\log K^*}}$$

where $K^*$ is the target mode count.

1. Training the large-scale FNO with the rescaled hyperparameters, obviating any further tuning.

Empirically, this maintains accuracy and stability while reducing HPO computational cost by factors of 3–10, with batch size and other Adam parameters also robustly transferring (Li et al., 24 Jun 2025).

5. Multiobjective and Large-Scale HPO in the Fourier Domain

In high-dimensional or multiobjective optimization tasks, Fourier domain HPO can incorporate complex objectives and large search spaces efficiently. For example, in FNO-based ocean modeling, DeepHyper leverages centralized Bayesian optimization with tree-based surrogates and $q$-batch Upper Confidence Bound (UCB) acquisition. The objective combines MSE and anomaly correlation (ACC) in a randomized scalarization to explore the Pareto front uniformly (Sun et al., 2024). Key hyperparameters include:

• Data preprocessing (e.g., padding strategies)
• FNO architecture (number of modes, blocks, channel widths, activation functions)
• Training strategies (optimizer type, learning rate, batch size, regularization)

Strong performance is achieved for moderate latent widths, high mode counts, and careful optimizer selection (e.g., AdamW with $lr \sim 10^{-3}$), with early stopping accelerating pruning of suboptimal configurations.

6. Limitations, Best Practices, and Scope

Fourier/spectral HPO methods exploit landscape structure—sparsity, low-degree polynomiality, or explicit frequency dependence—but may be less effective for objectives lacking such properties. For spectral surrogate methods, the choice of truncation degree and regularization requires careful tuning; in quantum approaches, device constraints limit $d$ and the achievable Fourier order (Consul-Pacareu et al., 2023, Hazan et al., 2017). For FNO μTransfer, the framework is sensitive to the sub-Gaussianity of stochastic gradient steps and requires proxy models with $K_{\text{proxy}}$ not too small ($K \geq 6$ recommended) (Li et al., 24 Jun 2025).

Practitioners are advised to:

• Initialize kernel weights and learning rates with μP scaling in FNO contexts.
• Use moderate proxy models (e.g., $K_{\text{proxy}}=6$–$12$) for transfer.
• Employ element-wise gradient clipping in operator layers to uphold sub-Gaussian assumptions.
• Restrict rescaling to μP-sensitive hyperparameters; leave others unchanged.
• Apply spectral surrogate modeling where known or suspected low-degree sparse dependence exists.

These methodologies remain most powerful where the optimization objective exhibits explicit or latent spectral compositionality, a characteristic widespread in contemporary scientific computing, machine learning, and signal processing applications.

7. Comparative Table: Core Approaches

Method/Domain	Spectral Representation	HPO Strategy / Key Algorithm	Reference
Multigrid PDEs (LFA-based)	Fourier symbol of error matrix	Minimax (ROBOBOA, HANSO-FI)	(Brown et al., 2020)
FNOs (μTransfer-FNO)	Kernel tensor in Fourier domain	μP scaling law with zero-shot HPO	(Li et al., 24 Jun 2025)
Neural networks (general)	Ortho. polynomial, Fourier exp.	Sparse recovery (Harmonica), Lasso	(Hazan et al., 2017)
Quantum ML	Angle-encoded Fourier series	Fourier-VQA plus classical search	(Consul-Pacareu et al., 2023)
Ocean modeling with FNO	FNO architecture, Fourier loss	Bayesian opt., Pareto front	(Sun et al., 2024)

These approaches demonstrate the breadth and potency of Fourier domain hyper-parameter optimization across scientific and machine learning domains, providing sample- and compute-efficient tuning in highly structured, high-dimensional settings.