Importance sampling for Bayesian inference: polynomial-dimension dependent error bounds

Abstract: Many Bayesian inference problems involve high-dimensional models where the performance of standard importance sampling (IS) methods often degrades rapidly as the dimensionality increases. Classical analyses of IS typically rely on the assumption that observations are arbitrary but fixed (i.e., deterministic), thereby neglecting the probabilistic structure that the Bayesian model induces on the data. In this paper, we adopt the perspective that observations are themselves random variables whose distribution is governed by the underlying model. Within this probabilistic framework, we identify a model-dependent function, referred to as the link function, which connects the fixed- and random-observation formulations. We provide a characterization of the $L^2$ Monte Carlo estimation error: specifically, we show that the $L^2$ error bounds are finite and converge at the standard Monte Carlo rate $O(N^{-1/2})$, for arbitrarily large dimension, if and only if the link function is Bochner integrable. This result reveals the fundamental quantity controlling the approximation error and establishes a mechanism to manage the dependence on the model state dimension. Consequently, our approach provides a principled way to alleviate the challenges of high dimensionality, offering insights that transcend worst-case analyses dominant in the existing literature. Finally, we derive explicit analytical examples of the dimensional scaling of the associated errors for several model classes, including linear-Gaussian systems and models with bounded observation functions.

• The paper establishes that Bochner integrability of the link function is essential for achieving the optimal Monte Carlo error rate of O(N⁻¹ᐟ²) in high-dimensional Bayesian inference.
• It provides explicit dimension-dependent error bounds and sample complexity estimates for models such as linear–Gaussian systems and those with bounded observation functions.
• The analysis bridges functional analysis with Bayesian methodology, offering practitioners practical criteria to assess and mitigate the curse of dimensionality in Monte Carlo sampling.

Dimension-Dependent Error Bounds for Importance Sampling in Bayesian Inference

Introduction

Importance sampling (IS) remains a fundamental Monte Carlo methodology for approximating expectations in Bayesian inference, especially for models where analytic integration is infeasible. Despite widespread application, IS performance can degrade rapidly in high-dimensional settings—a manifestation of the "curse of dimensionality." Classical error analyses largely assume fixed observed data, neglecting the Bayesian perspective where observations themselves are random variables governed by the model. This paper introduces a comprehensive probabilistic framework for IS error analysis that explicitly incorporates the randomness of observations and model structure, yielding polynomial and, in certain cases, uniform error bounds with respect to the state dimension. The analysis leads to new characterization theorems for $L^2$ error and sample complexity, identifies the critical role of Bochner integrability of a model-dependent link function, and provides explicit dimension-dependent error calculations for several model classes, including linear–Gaussian systems and models with bounded observation functions (2604.02094).

Probabilistic Framework and Importance Sampling Setup

The authors consider Bayesian models characterized by random variables $X \in \mathbb{R}^{d_x}$ (state of interest) and $Y \in \mathbb{R}^{d_y}$ (observation), specified with prior $\pi_0$ and observation model $g(y \mid x)$. The posterior law for fixed $Y=y$ is $\pi_y(x) \propto g(y|x) \pi_0(x)$. Standard importance sampling proceeds by drawing $N$ i.i.d. samples $x^i \sim \pi_0$, computing weights $w^i \propto g(y|x^i)$, and estimating posterior expectations via weighted sums.

In contrast with standard approaches assuming deterministic $X \in \mathbb{R}^{d_x}$0, the framework here treats $X \in \mathbb{R}^{d_x}$1 as a random variable. Thus, both the likelihood and posterior become random functionals, and the IS approximation inherits this randomness. The key technical object is the "link function" $X \in \mathbb{R}^{d_x}$2, which acts as a normalized density mapping between observation and state variables. The paper analyzes the Bochner integrability properties of the random element $X \in \mathbb{R}^{d_x}$3 in $X \in \mathbb{R}^{d_x}$4, which turns out to be pivotal for the error control in IS.

Main Results: Bochner Integrability and Error Bounds

The paper provides a sharp characterization of the $X \in \mathbb{R}^{d_x}$5 Monte Carlo error under random observations and Bayesian models:

The error of the standard IS estimator converges at rate $X \in \mathbb{R}^{d_x}$6, uniformly in $X \in \mathbb{R}^{d_x}$7, if and only if the link function $X \in \mathbb{R}^{d_x}$8 is Bochner square-integrable with respect to the joint law of the prior and the data-model-induced law of $X \in \mathbb{R}^{d_x}$9.

Explicitly, the $Y \in \mathbb{R}^{d_y}$0 expected error for bounded test function $Y \in \mathbb{R}^{d_y}$1 is bounded:

$Y \in \mathbb{R}^{d_y}$2

where $Y \in \mathbb{R}^{d_y}$3 is determined by the Bochner norm $Y \in \mathbb{R}^{d_y}$4. Importantly, the necessary and sufficient condition for finiteness and convergence of IS error is Bochner $Y \in \mathbb{R}^{d_y}$5 integrability of the link function.

The paper demonstrates that, for several practically relevant Bayesian model families, this condition can be checked via tractable, often one-dimensional, integrals. When Bochner integrability yields $Y \in \mathbb{R}^{d_y}$6 for a polynomial $Y \in \mathbb{R}^{d_y}$7, the sample complexity to reach a prescribed accuracy grows only polynomially with $Y \in \mathbb{R}^{d_y}$8; in particularly favorable cases ($Y \in \mathbb{R}^{d_y}$9), the error bound is uniform in $\pi_0$0.

Linear–Gaussian Models

For linear–Gaussian state-space models, the authors carry out all computations in closed form and show that $\pi_0$1 can be bounded independently of $\pi_0$2. Thus, the $\pi_0$3 error of IS decays as $\pi_0$4, regardless of the state-space dimension—contradicting the common perception that IS is always exponentially hard in high dimensions. The results can further be extended to varying observation dimension $\pi_0$5, with explicit conditions on the observation noise covariance ensuring error control.

Models with Bounded or Elliptically-Symmetric Likelihoods

For observation models arising from sensors with bounded dynamic range (bounded observation functions) or having elliptically symmetric likelihoods (including Gaussian, Laplace, Student-t, Cauchy, and others), the second moment $\pi_0$6 reduces to the boundedness of a one-dimensional radial integral depending on the observation map and noise profile. For bounded $\pi_0$7 and well-behaved noise tails, polynomial or even uniform dimension error scaling is obtained. Explicit dimension-dependent sample complexity formulas are derived for observation maps involving saturating nonlinearities coupled with moderate channel sensitivity.

Contrasts with Classical Worst-Case Analyses

Classical worst-case results (e.g., [chatterjee2018sample], [agapiou2017importance], [Rebeschini15]) typically establish exponential sample complexity in terms of the dimension, based on arbitrarily chosen target and proposal distributions or by fixing the data. Here, by leveraging the probabilistic structure of data generation, the paper shows that average-case error bounds may be polynomial or even uniform in $\pi_0$8 for structured Bayesian models.

This distinction is crucial: under canonical IS (prior as proposal), the model structure and the randomness of observations can provide concentration effects unaccounted for in worst-case calculations. The analysis justifies the performance of IS and related Monte Carlo schemes in certain large-scale Bayesian inference problems where exponential scaling is mitigated by the model's probabilistic architecture.

Implications, Extensions, and AI Perspectives

The theoretical framework provides practitioners with tractable integrability checks for IS stability in complex inference tasks, bypassing exhaustive simulation or pessimistic error bounds. In practical inference—such as signal processing, computational biology, and high-dimensional Bayesian inverse problems—this perspective clarifies when standard IS can remain viable, and suggests structural model properties to target when designing Bayesian algorithms.

The results inform the development of adaptive IS and SMC schemes in large-scale models. Further, the identification of Bochner integrability as controlling error scaling for random observation models suggests use in analyzing and benchmarking advanced stochastic inference methods in high-dimensional AI pipelines, including deep generative Bayesian learning and probabilistic programming.

The Bochner-functional analytic approach is potentially extensible to more general function spaces, observation processes, and to the analysis of subsampling and distributed inference protocols, which are highly relevant for modern scalable Bayesian machine learning.

Conclusion

This paper establishes an intrinsic probabilistic criterion—Bochner integrability of the link function—for controlling the $\pi_0$9 error of importance sampling in Bayesian inference with random observations. This condition yields polynomial or uniform sample complexity with respect to model dimension in structured model classes, including linear–Gaussian and bounded observation settings. The analysis bridges abstract functional analysis with concrete diagnostic criteria for practitioners, substantiates the potential for IS in certain high-dimensional inference problems, and delineates the true boundary between deterministic worst-case and probabilistically-structured Bayesian analyses (2604.02094).