Bayes@N: Linking NML and Bayesian Inference

• Bayes@N is a principle where the normalized maximum likelihood (NML) distribution exactly matches Bayesian predictive and marginal distributions using a sample-size-specific prior.
• It guarantees minimax coding regret and finite-sample Bayesian coherence, enabling optimal sequential prediction and efficient computation.
• By leveraging sufficient statistics and grid-based weights, Bayes@N unites MDL principles with Bayesian inference, offering both genuine mixtures and valid signed priors.

Bayes@N denotes the principle that, for every fixed sample size $N$, the normalized maximum likelihood (NML) distribution associated with a statistical model coincides exactly with the Bayesian predictive and marginal distributions for that sample size, under a specific (possibly signed) prior $w_N$. This property was established for arbitrary parametric families of i.i.d. distributions, providing a conceptual link between Bayesian inference and the minimum description length (MDL) paradigm in universal coding, gambling, and prediction. Bayes@N guarantees minimax regret while delivering finite-sample Bayesian coherence (such as conditionalization and exchangeability of updates), regardless of sample size. Notably, for certain models, the corresponding prior can be chosen nonnegative, yielding a genuine Bayes mixture (Barron et al., 2014).

1. Core Definitions and Mathematical Foundations

Consider a statistical model specified by i.i.d. densities or mass functions $f(x^n|\theta) = \prod_{i=1}^n f(x_i|\theta)$, with parameter $\theta$ belonging to a parameter space $\Theta$ and sample space $\mathcal{X}$. The normalized maximum likelihood distribution is defined as

$P_\text{NML}(x^n) = \frac{m(x^n)}{C_n},$

where $m(x^n) = \max_\theta f(x^n|\theta)$ and $C_n = \sum_{y^n \in \mathcal{X}^n} m(y^n)$ (or the analogous integral for continuous spaces). Shtarkov’s theorem ensures that $P_\text{NML}$ uniquely achieves the minimax pointwise regret

$$\min_q \max_{x^n} \left[-\log q(x^n) + \log m(x^n)\right]$$

for coding and prediction.

The principle Bayes@N asserts that for each $n$, there exists a prior measure $w_n$ (potentially signed) such that

$$P_\text{NML}(x^n) = \int_\Theta w_n(\theta)\;f(x^n|\theta)\;d\theta,$$

and for any data prefix $x^i$,

$$P_\text{NML}(x^i) = \sum_{k=1}^M W_{k,n} f(x^i|\theta_k)/C_n,$$

where $W_{k,n}$ are explicit weights derived via a linear system involving sufficient statistics (Barron et al., 2014).

2. Mixture Representation of NML and the “Bayes@N Prior”

The Bayes@N mixture representation exploits sufficient statistics $T(x^n)$ taking $M$ distinct values. For an appropriately chosen $M$-point grid $\theta_1,\dots,\theta_M$ with independent model vectors $p_T(\cdot|\theta_k)$, the maximized distribution $m_T$ is decomposed by solving

$m_T(t) = \sum_{k=1}^M p_T(t|\theta_k) W_{k,n},$

yielding a discrete prior $$w_n(\theta) = \sum_{k=1}^M W_{k,n}\delta_{\theta_k}(\theta)$$. The condition for a genuine Bayesian mixture ($w_n \ge 0$) is that $m_T$ lies in the convex hull of $\{p_T(\cdot|\theta_k)\}$; if not, $w_n$ exhibits negative parts, but the predictive distribution remains valid and nonnegative.

Illustrative examples: For Bernoulli($\theta$), $M=n+1$ and the grid $\theta_k=k/n$ with uniform or arcsine spacing recovers $W_{k,n}$ by matrix inversion. In the Gaussian location model, weights $W_{k,n}$ are determined via deconvolution over suitable grids of the mean parameter.

3. Bayesian Predictives, Minimax Regret, and Computational Efficiency

By construction,

$$P_\text{NML}(x_i|x^{i-1}) = P_\text{NML}(x^i)/P_\text{NML}(x^{i-1})$$

coincides with the Bayesian predictive under $w_n$. This duality ensures that at sample size $N$, the predictive updates are exactly Bayesian, conditionalization holds, and the sequence of updates is exchangeable.

Computationally, marginals and predictives require only $O(M)$ time per forward step once weights $W_{k,n}$ and model evaluations are cached, and the full weight solution is $O(M^3)$ (e.g., $O(n^3)$ for Bernoulli). Fast algebraic techniques (e.g., Vandermonde or Hankel structure for binomial models) further reduce this cost to $O(n^2)$ or $O(n\,\log^2 n)$. A direct enumeration for $|\mathcal{X}|^{n-i}$ continuations is infeasible beyond moderate $n$ (Barron et al., 2014).

4. Guarantees: Minimax Coding and Exact Bayesian Coherence

The mixture representation provides two fundamental guarantees:

• The codelength $-\log P_\text{NML}(x^n)$ equals the minimax worst-case regret, i.e., it never exceeds by more than $\log C_n$ the code length of the best parameter $\theta^*$ for any data sequence.
• The one-step predictives (for each horizon $N$) coincide precisely with Bayesian inference under $w_n$, with all conditionalization and finite-sample Bayesian properties intact.

A plausible implication is that statistical inference, coding, and sequential prediction can be interpreted simultaneously as both minimax-optimal and Bayesian (horizon-dependent) for arbitrary finite samples.

5. Relationship to MDL and Bayesian Modeling

Bayes@N resolves a long-standing question about the connection between the minimum description length (MDL) principle and Bayesian prediction. The existence of the finite-sample prior $w_N$ (even if signed) shows that NML’s minimax regret procedure is, at every $N$, a fully Bayesian predictive with a particular, sample-size-specific prior. When $w_N$ is positive, all consequences of Bayesian modeling (coherence, valid posterior probabilities, etc.) follow identically; when $w_N$ is signed, the mixture remains algebraically and algorithmically valid for predictive purposes but loses the interpretation of uncertainty over parameters.

6. Extensions: Examples and Practical Implications

For Bernoulli and multinomial models, $M$ is polynomial in $n$; also, arcsine grids allow nonnegative $w_n$ and thus true Bayes mixtures. In Gaussian location families, symmetrical grids may sometimes yield nonnegative weights when intervals are sufficiently broad.

The property Bayes@N enables significantly faster computations of marginals and conditionals by reducing high-dimensional sums to algebraic evaluations using the precomputed prior, providing scalable algorithms for universal coding and prediction consistent both with MDL and finite-sample Bayesian reasoning.

7. Limitations, Signed Priors, and Generalizations

If the mixture weights $w_n$ cannot be chosen nonnegative, the solution defaults to signed priors, which lack interpretation as probability measures over parameters but suffice for algebraic and operational validity. The necessary condition for a positive mixture is model-dependent and relates to the geometry of sufficient statistics and model vectors.

A plausible implication is that signed priors may be unavoidable in highly regularized or under-determined model classes, but for many practical models (one-parameter exponential families, large-grid multinomials, suitably chosen grids), honest positive priors are feasible, permitting Bayes@N to blend Bayesian inference and minimax regret seamlessly.

The central result is that, for every $N$, NML distributions are Bayesian under a specifically constructed sample-size-dependent prior $w_N$, yielding both rigorous minimax coding and Bayesian predictives, and enabling efficient computation of posterior marginals and conditionals with exact guarantees (Barron et al., 2014).