Bayesian Aggregation

Abstract: A general challenge in statistics is prediction in the presence of multiple candidate models or learning algorithms. Model aggregation tries to combine all predictive distributions from individual models, which is more stable and flexible than single model selection. In this article we describe when and how to aggregate models under the lens of Bayesian decision theory. Among two widely used methods, Bayesian model averaging (BMA) and Bayesian stacking, we compare their predictive performance, and review their theoretical optimality, probabilistic interpretation, practical implementation, and extensions in complex models.