On Cold Posteriors of Probabilistic Neural Networks: Understanding the Cold Posterior Effect and A New Way to Learn Cold Posteriors with Tight Generalization Guarantees

Abstract: Bayesian inference provides a principled probabilistic framework for quantifying uncertainty by updating beliefs based on prior knowledge and observed data through Bayes' theorem. In Bayesian deep learning, neural network weights are treated as random variables with prior distributions, allowing for a probabilistic interpretation and quantification of predictive uncertainty. However, Bayesian methods lack theoretical generalization guarantees for unseen data. PAC-Bayesian analysis addresses this limitation by offering a frequentist framework to derive generalization bounds for randomized predictors, thereby certifying the reliability of Bayesian methods in machine learning. Temperature $T$, or inverse-temperature $\lambda = \frac{1}{T}$, originally from statistical mechanics in physics, naturally arises in various areas of statistical inference, including Bayesian inference and PAC-Bayesian analysis. In Bayesian inference, when $T < 1$ (cold'' posteriors), the likelihood is up-weighted, resulting in a sharper posterior distribution. Conversely, when $T > 1$ (warm'' posteriors), the likelihood is down-weighted, leading to a more diffuse posterior distribution. By balancing the influence of observed data and prior regularization, temperature adjustments can address issues of underfitting or overfitting in Bayesian models, bringing improved predictive performance.

• The paper demonstrates that cold posterior tempering rectifies prior-likelihood mismatches, improving predictive accuracy in Bayesian neural networks.
• It introduces Recursive PAC-Bayes, a novel method that efficiently updates sequential confidence measures and outperforms traditional PAC-Bayesian techniques.
• The paper further refines ELBO decomposition for mean-field variational models, offering deeper insights to enhance optimization and generalization.

Essay on "On Cold Posteriors of Probabilistic Neural Networks" by Yijie Zhang

Yijie Zhang’s thesis titled "On Cold Posteriors of Probabilistic Neural Networks" explores the intriguing phenomenon of cold posterior effects (CPE) within Bayesian deep learning frameworks. The central theme investigates how tempered posteriors with a temperature parameter $T < 1$ can sometimes outperform standard Bayesian posteriors, suggesting insights for both theoretical expansions and practical applications in machine learning.

Theoretical Insights and Contributions

Zhang’s research begins with a comprehensive analysis of Bayesian inference mechanisms, particularly focused on the nuances of posterior tempering. By employing a temperature parameter, denoted typically as $\lambda = 1/T$, the thesis explores how altering this parameter influences the predictive performance of probabilistic neural networks. Notably, a central claim made is the connection between the occurrence of CPE and model underfitting due to misspecification of either the prior or the likelihood. This is a pivotal clarity in understanding why CPE emerges and offers a nuanced interpretation beyond mere hyperparameter tuning.

Zhang extends the argument that cold posteriors are not merely beneficial artifacts but are genuinely valid Bayesian posteriors with different priors and likelihood configurations. This is evidenced through rigorous theoretical derivations and empirical validations, as discussed in Chapters 2 and 3, which reveal that such tempered posteriors represent alternative Bayesian settings that rectify initial mismatches between model assumptions and actual data distributions.

Empirical Evaluations

The thesis does not shy away from empirical analyses. It meticulously demonstrates through Bayesian linear regression and neural network experiments how exact inference can exhibit CPE, hence debunking any claims that CPE is merely a byproduct of approximate inference methods. The findings underscore that underfitting—and not necessarily computational approximations—plays a critical role in CPE manifestation. This insight is fortified through extensive data augmentations and neural network models where tempered posteriors outperform traditional setups.

Recursive PAC-Bayes Contribution

An integral part of Zhang’s work is the development of Recursive PAC-Bayes (RPB). This novel methodological approach tackles limitations in conventional PAC-Bayesian analysis, chiefly the inability to retain confidence information across sequential prior updates. The RPB framework ingeniously decomposes expected losses, thereby enabling efficient sequential updating which leverages the entirety of the available data, not just a subset, preserving vital confidence measures. This procedural innovation is empirically validated to outperform extant techniques, marking a significant breakthrough for Bayesian model evaluations.

Connections to Practical Techniques

Further, in an effort to link Recursive PAC-Bayes with other successful machine learning strategies, Zhang elucidates connections to cold posteriors and KL-annealing techniques. These relationships provide additional perspectives on RPB’s effectiveness, suggesting that the tempered, recursive approach aligns naturally with these practical methods, offering potential improvements in training dynamics and generalization capabilities.

ELBO Decomposition

Lastly, Zhang proposes a new decomposition of the Evidence Lower Bound (ELBO) for mean-field variational global latent variable models. This contribution offers a refined lens to examine the training processes of these models, enabling deeper insights into optimization and posterior evaluation—a valuable tool for researchers aiming to better control and comprehend model behaviors in varied applications.

Conclusion

Yijie Zhang’s thesis stands as a substantial academic endeavor that pushes the boundaries of our understanding of Bayesian inference in deep learning. By delineating the intricacies of cold posteriors and refining PAC-Bayesian analysis, the work provides both theoretical propulsion and pragmatic solutions for improved model performance and reliability. This research is poised to have lasting impacts, directing future explorations in Bayesian methodologies amidst the evolving landscape of artificial intelligence.