Gaussian Process-based learning with new MCMC-based implementation of Wishart prior on correlation matrix

Abstract: In probabilstic supervised learning of an input-output relationship - as a sample function of a Gaussian Process (GP) - priors are typically specified for the hyperparameters of the kernel that parametrises the covariance function of the GP, where the induced covariance matrix of the (resulting multivariate Normal) likelihood, governs the learning and prediction. When the sought function is highly multivariate, multiple lengthscale parameters must be learnt simultaneously, making inference difficult. We develop a ``self-assembled'' Wishart prior for the covariance matrix, while undertaking Bayesian inference on the kernel hyperparameters using MCMC. The construction uses a look-back window over recent MCMC iterations to define a time-step dependent scale matrix, thereby introducing adaptiveness to the chain. Results suggest that direct prior specification on the covariance matrix can be useful for diagnosing weakly informative inputs within the GP-based learning paradigm. We support our prior development with two distinct empirical illustrations - one on synthetic data, and another on a real-world dataset.

• The paper introduces an adaptive Wishart prior constructed during MCMC to improve the conditioning of covariance matrices in Bayesian GP models.
• Synthetic experiments demonstrate that inflating irrelevant input lengthscales yields up to a 50% RMSE reduction compared to traditional priors.
• The approach also serves as a diagnostic tool for variable selection, facilitating effective model reduction and superior predictive performance on real data.

Gaussian Process-based Learning with an MCMC-Constructed Wishart Prior on the Covariance Matrix

Motivation and Methodological Contributions

The paper presents a methodological innovation in Bayesian Gaussian Process (GP) learning, focusing on prior specification directly at the level of the kernel-induced covariance matrix via a Wishart prior whose scale matrix is adaptively constructed during MCMC inference (2605.27093). The motivation is grounded in the observation that standard GP practice relies on placing priors on individual kernel hyperparameters (e.g., lengthscales, amplitude, noise). While this approach is structurally compatible with kernel-based inference, it does not guarantee desirable properties for the resulting covariance matrix, especially in high-dimensional ARD contexts where many inputs may be only weakly informative. Poorly conditioned or near-singular covariance matrices—originating from marginal priors—present practical and inferential challenges.

The core contribution is a "self-assembled" Wishart prior: during MCMC, each proposal's kernel hyperparameters induce a covariance matrix, and the Wishart prior evaluates this matrix using a scale matrix constructed from a look-back window of previously accepted hyperparameter configurations. This construction ensures the prior acts adaptively, favoring covariance structures recently validated by the chain. The approach stays within the kernel-parametric regime, eschewing unconstrained empirical covariance estimation in favor of an interpretably Bayesian, adaptive prior mechanism.

Formal Prior Construction and Implementation

Let $\boldsymbol{\theta}$ denote the kernel hyperparameter vector and $\Sigma_{\boldsymbol{\theta}}$ the covariance matrix induced at $p$ training points. In iteration $t$ of the MCMC chain, the scale matrix $\mathbf{V}_t$ is constructed using the mean hyperparameter vector over a look-back window of length $n$. The Wishart prior is then evaluated at the proposal's covariance matrix $\Sigma_t^{(*)}$:

$$\pi_W\!\left(\Sigma_t^{(*)} \mid \mathbf{V}_t\right) \propto |\Sigma_t^{(*)}|^{\frac{n-p-1}{2}} |\mathbf{V}_t|^{-\frac{n}{2}} \exp\left(-\frac{1}{2}\operatorname{tr}(\mathbf{V}_t^{-1}\Sigma_t^{(*)})\right)$$

This prior penalizes proposals yielding nearly singular matrices (via the determinant term) and rewards proximity to typical recent covariance structures (via the trace term). Posterior inference incorporates this prior in the Metropolis-Hastings acceptance probability, yielding an adaptive chain whose target varies with $\mathbf{V}_t$. The authors demonstrate that the adaptation is diminishing and the chain is contained, thus guaranteeing MCMC convergence.

Synthetic Experiments: Relevance Separation and Predictive Performance

Extensive synthetic studies in a 15-dimensional input space (with 3–12 relevant variables per scenario) illustrate the strong mechanistic implications of the Wishart prior as compared to conventional Normal priors and combined Wishart + Normal priors. The experiments include both interaction-heavy and structurally simpler generative settings and are benchmarked against deep neural networks (DNNs).

In all synthetic scenarios, GP models with the Wishart prior demonstrate markedly stronger input relevance separation, as quantified by the ratio of mean relevant input lengthscale to mean irrelevant input lengthscale (values $\ll 1$ under the Wishart prior). This effect is not a consequence of dramatically shrinking relevant-input lengthscales; rather, it is predominately due to substantial inflation of irrelevant input lengthscales:

Figure 1: Baseline 6-relevant-input case, showing decisive lengthscale separation in the Wishart prior.

Predictive accuracy, as measured by RMSE and MAE, improves correspondingly. Notably, the Wishart prior yields RMSE reductions of approximately 50% relative to competing GP specifications in sparse scenarios; improvements persist when the problem is less sparse or structurally simpler. DNN benchmarks, even when trained on much larger data, do not replicate these gains, despite recovering appropriate variable importance rankings via permutation diagnostics. The Wishart prior confers predictive benefit not simply via relevance identification, but through more aggressive suppression of nuisance directions at the covariance level.

Empirical Application: Tetouan City Energy Data

Applying the methodology to the Tetouan City energy dataset, the authors analyze both the full 5-input model and a reduced 4-input variant (the latter motivated by diagnostic insights from the Wishart prior). In the full 5-input setting, the Wishart prior achieves the lowest RMSE and MAE, and the highest 95% empirical coverage among all specifications. Lengthscale traceplots exhibit one input being sharply downweighted by the covariance-level prior, indicating a weak contribution under ARD-squared-exponential smoothness assumptions.

Subsequent removal of this input yields improved predictive performance across all specifications, with the combined Wishart + Normal prior outperforming the Wishart-only prior in the reduced-input regime. This demonstrates the practical value of adaptive covariance-level prior structure—not only as a mechanism for improved prediction, but also as a diagnostic for input relevance and effective dimension reduction.

Figure 2: Normal prior post-burn-in lengthscale chains, highlighting lack of aggressive irrelevant input suppression.

Figure 3: Normal prior predictive summary, illustrating moderate fit to test data.

Figure 4: Normal prior predicted mean versus true test response, showing moderate but not optimal alignment.

Figure 5: Normal prior lengthscale chains in reduced-input analysis, with improved concentration.

Figure 6: Normal prior predictions in reduced-input analysis, exhibiting higher RMSE improvement.

Figure 7: Wishart prior lengthscale chains in full-input analysis, showcasing aggressive suppression of a weak input.

Figure 8: Wishart prior predictive summary in full-input Tetouan analysis, achieving best fit to high-response regime.

Theoretical and Practical Implications

The empirical and methodological findings underscore several important theoretical implications:

• Covariance-level prior specification is distinct from hyperparameter-level priors: The Wishart prior acts directly at the level determining likelihood and numerical behavior, and is agnostic to scalar prior targets for individual hyperparameters.
• Adaptive self-assembly of the scale matrix links recent chain history to prior structure: This approach attunes the prior to realistically plausible regions of covariance space, enhancing stability and relevance separation.
• Relevance separation is effected not by shrinking relevant-input lengthscales, but by inflating those corresponding to nuisances: Posterior inference under the Wishart prior exhibits strong suppression of non-informative dimensions, even in weakly identified regions.
• Practical model reduction and diagnostic functionality: In real-world data settings, the prior can serve as an effective diagnostic, identifying weakly informative inputs and motivating reduced models.

Potential for future research includes further generalization (different kernel classes, larger $\Sigma_{\boldsymbol{\theta}}$0), refined adaptive scale constructions (broader window strategies, different weighting), and applications to structural variable selection where input redundancy or irrelevance is prominent.

Conclusion

The paper demonstrates that an MCMC-constructed Wishart prior on the kernel-induced covariance matrix is a powerful tool for Bayesian GP learning in high-dimensional ARD regimes. By acting directly at the level governing likelihood and prediction, and adaptively aligning prior structure to chain history, the method offers sharper input relevance separation and improved predictive accuracy. Empirical results in both synthetic and real-data contexts reveal that covariance-level prior specification is a valuable complement to conventional hyperparameter priors, particularly for nuisance input suppression and practical variable selection. The approach has substantial potential for more effective GP inference, model reduction, and interpretability in multi-input supervised learning tasks.