Infinite-Dimensional Priors

• Infinite-dimensional priors are probability measures defined on spaces like separable Banach and Hilbert spaces, constructed via orthonormal expansions and projective limits.
• They are rigorously built with specific moment and decay conditions to ensure almost-sure convergence and regularity in Bayesian inference for ill-posed inverse problems.
• These priors, ranging from Gaussian and non-Gaussian forms to neural network Gaussian processes, facilitate robust uncertainty quantification and function-space learning.

A prior in infinite dimensions is a probability measure defined on an infinite-dimensional space, most commonly a separable Banach or Hilbert space, or on spaces of distributions or functions. These priors play a foundational role in Bayesian statistical inference for ill-posed inverse problems, nonparametric Bayesian estimation, and function-space learning models where the parameter of interest is not finite-dimensional. The construction, theoretical properties, and computational handling of such priors require rigorous mathematical frameworks and specialized algorithms to address unique challenges arising from infinite dimensionality, lack of Lebesgue measure, and topology of the underlying space.

1. Construction of Priors in Infinite Dimensions

A canonical approach to constructing priors on infinite-dimensional Banach or Hilbert spaces uses countable expansions in terms of orthonormal or Schauder bases. A “product prior” on a space $X$ with basis $\{x_k\}$ is specified as

$u = \sum_{k=1}^\infty \gamma_k\,\xi_k\,x_k,$

where $(\gamma_k)$ encodes basis-dependent scaling, and $\{\xi_k\}$ are i.i.d. random variables with specified marginal law on $\mathbb{R}$ (Hosseini, 2018, Hosseini, 2016). Moment and decay conditions on $(\gamma_k)$ and the law of $\xi_k$ control almost-sure convergence and regularity. Gaussian measures are defined by mean vector and trace-class covariance operator on the Hilbert space (Alexanderian et al., 2014).

On the space $M(V)$ of Borel probability measures over a Polish space $V$, any prior is uniquely determined by its consistent finite-dimensional marginals via a projective limit construction. For each finite partition $I$, select a measure $\mu_I$ on the simplex $\Delta_I$; compatibility under coarser-to-finer partitions (projectivity), and a first-moment condition, ensure existence and uniqueness of a Radon law on $M(V)$ (Orbanz, 2011). Dirichlet and Pitman–Yor processes arise as specific projective families.

Neural network priors in the infinite-width regime are induced in function space as the weak limit of networks as widths tend to infinity. The resulting object is a Gaussian process (NNGP) prior with analytically characterized kernel (Adlam et al., 2020).

2. Classes and Examples of Infinite-Dimensional Priors

Infinite-dimensional priors can be classified by analytic and geometric properties of their defining laws:

• Gaussian priors: Defined by zero-mean and covariance operators; canonical in linear Bayesian inverse problems due to analytic tractability (Alexanderian et al., 2014).
• Non-Gaussian product priors: Instances include:
  • ℓₚ and G_{p,q} priors: Product measures with densities $f_{\ell_p}(t)\propto \exp(-|t|^p)$; $p<1$ yields heavy-tailed, non-convex, and infinitely-divisible laws (Hosseini, 2016).
  • Bessel-K priors: Generalize the gamma to infinite dimensions; characterized by Bessel function densities, interpolating between Laplace, gamma, and besov-like priors. They model compressibility/sparsity (Hosseini, 2018).
  • Dirichlet and related processes: Priors on spaces of distributions, constructed by projective limits of Dirichlet marginals or via group invariance for noninformative settings (Orbanz, 2011, Terenin et al., 2017).
  • Pure jump Lévy process priors: Support on bounded-variation (BV) function spaces, constructed using compound Poisson or stable laws (Hosseini, 2016).

A key feature distinguishing these priors is their tail decay and associated regularization effect in inverse problems. Infinitely-divisible (ID) laws and Lévy–Khintchine representations are central in encoding heavy tails and sparsity-promoting behavior.

3. Theoretical Properties: Well-Posedness, Invariance, and Support

Well-posedness of Bayesian inference with infinite-dimensional priors requires delicate interplay between prior tails and the likelihood. For general priors μ₀ and forward map G, under broad integrability and Lipschitz conditions on the negative log-likelihood Φ(u; y), the posterior

$d\mu^y/d\mu_0(u)\propto\exp(-\Phi(u;y))$

is guaranteed to exist, be unique, and depend continuously on y under total-variation and Hellinger metrics, provided suitable moment and integrability conditions on μ₀ (Hosseini, 2016, Hosseini, 2018). For Gaussian priors, trace-class covariance ensures existence of the Radon–Nikodym derivative and all classic Bayesian formulas extend with Fredholm determinants and trace operators replacing determinants and traces (Alexanderian et al., 2014).

Noninformative priors in infinite-dimensional distribution spaces arise as invariants under large transformation groups. Requiring prior invariance under transformations of finite partitions leads uniquely, via projective/moment consistency, to the improper Dirichlet Process (DP(0)), or, for any small ε, to the proper Dirichlet process DP(ε, H) (Terenin et al., 2017). This construction is optimal in the sense that, combined with an exchangeable likelihood, it yields a unique posterior.

For practical and theoretical flexibility, frameworks such as Marginally Specified Priors (MSP) decompose the prior into informative marginal laws over functionals (finite-dimensional) and a nonparametric conditional prior, yielding a class of infinite-dimensional priors that respect both genuine prior information and nonparametric flexibility (Kessler et al., 2012).

4. Markov Chain Monte Carlo and Computational Techniques

Sampling posteriors in infinite-dimensional spaces requires algorithms with dimension-robust mixing and computational guarantees. Key developments include:

• Prior-reversible proposal kernels: For infinite-dimensional product priors (including non-Gaussian/Bessel-K), autoregressive kernels designed to be reversible with respect to the prior law allow the construction of RCAR and pCN-type Metropolis–Hastings samplers whose mixing times and Monte Carlo error bounds do not degrade with increasing discretization dimension (Hosseini, 2018, Hosseini et al., 2018, Vollmer, 2013). For product priors, dimension-independent spectral gaps ensure bounded asymptotic variance and MSE for pathwise averages (Vollmer, 2013, Hosseini et al., 2018).
• Transport and Wasserstein-like semimetrics: Ergodicity and spectral gaps can be established in weighted Wasserstein-type semimetrics, leveraging Lyapunov functions, even for non-Gaussian infinite-dimensional priors. Perturbation-theoretic error bounds yield explicit control of discretization and prior-approximation errors (Hosseini et al., 2018).
• Neural network Gaussian process limits: Infinite-width NNs enable exact GP priors, for which Bayesian marginalization and classification are implementable using sampling-based/analytic methods for the GP posterior (Adlam et al., 2020).
• Marginally specified priors: MCMC samplers for classical nonparametric priors (e.g., Dirichlet process mixtures) can be adapted with simple MH ratio adjustments for MSP priors, preserving computational tractability (Kessler et al., 2012).

5. Applications in Bayesian Inverse Problems and Nonparametrics

Infinite-dimensional priors underpin a broad spectrum of applications:

• Bayesian inverse problems: Gaussian, sparse/product, and heavy-tailed priors yield well-posed inference for PDE-governed and functional parameter recovery problems; error control for finite-mode (Galerkin, spectral) approximations is explicit and dimension-robust (Alexanderian et al., 2014, Hosseini, 2018, Hosseini, 2016, Hosseini et al., 2018).
• Nonparametric Bayes: Priors such as Dirichlet processes, Pitman–Yor processes, and related projective limit constructions admit principled modeling of probability densities and cumulative distribution functions; group-invariance approaches uniquely specify noninformative priors (Orbanz, 2011, Terenin et al., 2017).
• Function-space learning: NNGP priors for infinitely-wide neural nets formalize uncertainty quantification and calibration for classification/regression tasks, including transfer learning extensions via infinite last-layer heads (Adlam et al., 2020).
• Marginal constraints: MSPs enable flexibly combining informative finite-dimensional prior beliefs with standard infinite-dimensional nonparametric priors, with applications to density estimation and large sparse contingency table analysis (Kessler et al., 2012).

6. Stability, Approximation, and Practical Implications

Bayesian procedures with infinite-dimensional priors exhibit stability to prior and likelihood perturbations under explicit conditions. Key results include:

• Posterior stability under prior perturbation: For priors defined by truncated or approximated basis expansions, explicit transport semimetric bounds quantify how errors propagate to posteriors, circumventing mutual singularity obstacles (Hosseini et al., 2018).
• Finite-mode and numerical approximation: Projection methods produce approximate posteriors $\mu_N^y$ that converge in total variation and Hellinger distance to the infinite-dimensional posterior at rates determined by operator norms of the projection error (Alexanderian et al., 2014, Hosseini et al., 2018, Hosseini, 2016).
• Moment and tail controls: Well-posedness is closely tied to the integrability of sub-multiplicative moment functions against the prior law, particularly for ID and heavy-tailed priors (Hosseini, 2016).
• Intersection with computational practice: Existence theorems for projective-limit priors and other infinite-dimensional constructions provide guidance for algorithmic design but may not yield practical samplers directly; conjugacy and stick-breaking methods are critical for computation (Orbanz, 2011).

7. Summary Table: Principal Infinite-Dimensional Priors and Properties

Prior Class	Defining Data	Topological/Measure Conditions
Gaussian	Mean $m$, Covariance $C$	Separable Hilbert, $C$ trace-class, positive-definite
Product (ℓₚ, G_{p,q}, etc.)	IID law for coordinates	Basis scaling $(\gamma_k)\in\ell^2$, moments/ID property
Bessel-K	Shape $p$, scale $\sigma$	Closure under convolution; compressive for small $p$
Dirichlet Process	Concentration $\alpha$, base $H$	Projective marginals; mean in $M(V)$; Polish space
Pure Jump Lévy/BV	Poisson rate/jump law	Support on $BV$ via tightness, Helly’s theorem
Neural Network GP (NNGP)	Network kernel parameters	Infinite-width limit; function space parametrization
Marginally Specified Prior	Canonical prior, marginal law	Preservation of support/consistency; implemented via adjustment

This table highlights the analytic assumptions and necessary topological structure for existence and tractability of infinite-dimensional priors, central to infinite-dimensional Bayesian inference across domains (Alexanderian et al., 2014, Orbanz, 2011, Hosseini, 2018, Adlam et al., 2020, Hosseini et al., 2018, Hosseini, 2016, Terenin et al., 2017, Kessler et al., 2012, Vollmer, 2013).