On Uniform, Bayesian, and PAC-Bayesian Deep Ensembles

Abstract: It is common practice to combine deep neural networks into ensembles. These deep ensembles can benefit from the cancellation of errors effect: Errors by ensemble members may average out, leading to better generalization performance than each individual network. Bayesian neural networks learn a posterior distribution over model parameters, and sampling and weighting networks according to this posterior yields an ensemble model referred to as a Bayes ensemble. This study reviews the argument that neither the sampling nor the weighting in Bayes ensembles are particularly well suited for increasing generalization performance, as they do not support the cancellation of errors effect. In contrast, we show that a weighted average of models, where the weights are optimized by minimizing a second-order PAC-Bayesian generalization bound, can improve generalization. It is crucial that the optimization takes correlations between models into account. This can be achieved by minimizing the tandem loss, which requires hold-out data for estimating error correlations. The tandem loss based PAC-Bayesian weighting increases robustness against correlated models and models with lower performance in an ensemble. This allows us to safely add several models from the same learning process to an ensemble, instead of using early-stopping for selecting a single weight configuration. Our experiments provide further evidence that state-of-the-art intricate Bayes ensembles do not outperform simple uniformly weighted deep ensembles in terms of classification accuracy. Additionally, we show that these Bayes ensembles cannot match the performance of deep ensembles weighted by optimizing the tandem loss, which additionally provides nonvacuous rigorous generalization guarantees.

• The paper demonstrates that traditional Bayesian ensembles struggle with error cancellation, often underperforming compared to simple uniformly-weighted ensembles.
• It shows that PAC-Bayesian optimization using the tandem loss improves ensemble weighting and provides robust generalization guarantees.
• The study emphasizes a practical shift toward well-weighted deep ensembles, encouraging methods that balance model diversity and performance.

Bayesian vs. PAC-Bayesian Deep Neural Network Ensembles

The research paper "Bayesian vs. PAC-Bayesian Deep Neural Network Ensembles" by Nick Hauptvogel and Christian Igel examines the efficacy of Bayesian and PAC-Bayesian methods in constructing deep neural network ensembles. The core contention is that traditional Bayesian ensemble methods may not effectively harness the ensemble effect to boost generalization performance, unlike PAC-Bayesian approaches optimized for this purpose.

Summary of Key Concepts

The paper tackles the distinction between Bayesian approaches and PAC-Bayesian methods when applied to deep neural network ensembles. Bayesian neural networks (BNNs) inherently tackle epistemic uncertainty by estimating a posterior distribution over model parameters. The resulting Bayes ensemble incorporates this posterior to produce predictions. In contrast, PAC-Bayesian methods leverage a distribution designed to minimize specific generalization bounds—in this case, the tandem loss, which considers pairwise error correlations between models in the ensemble.

Bayesian Ensembles and Generalization

The authors identify a fundamental limitation in Bayesian ensembles. Specifically, they argue that the Bayes ensemble’s sampling and weighting do not naturally facilitate error cancellation among ensemble members. This behavior is elucidated via the Bernstein-von Mises theorem, which demonstrates that Bayesian model averaging (BMA) converges to the maximum likelihood estimate as dataset size increases, limiting the beneficial diversity among ensemble models. Empirical evaluations underscored that simple, uniformly-weighted deep ensembles often outperform computationally expensive Bayesian ensemble methods across various datasets.

PAC-Bayesian Optimization

PAC-Bayesian methods offer an alternative, aiming to improve ensemble performance by optimizing the ensemble member weighting through minimization of a PAC-Bayesian generalization bound. These methods take advantage of the tandem loss to account for correlations between models, thus preserving ensemble diversity and improving generalization performance. Notably, this approach requires hold-out data for accurate error correlation estimation, which constitutes a constraint for scenarios with limited data.

Experimental Evaluation

Empirical evaluations were conducted across multiple datasets: IMDB, CIFAR-10, CIFAR-100, and EyePACS, employing neural network architectures such as CNN LSTMs and various ResNets. The results indicate that:

1. Simple Uniformly-Weighted Ensembles: These ensembles matched or exceeded the performance of state-of-the-art Bayesian ensembles from literature, questioning the added complexity and computational cost of Bayesian approaches for generalization performance enhancement.
2. PAC-Bayesian Weighted Ensembles: Implementing PAC-Bayesian weighted deep ensembles using the tandem loss bound showed comparable or slightly superior performance vis-à-vis their uniformly-weighted counterparts. Moreover, the PAC-Bayesian approach provided rigorous, non-vacuous generalization guarantees.
3. Snapshot Ensembles (SSE): When intermediate models (snapshots) from the same training run were included and weighted via PAC-Bayesian optimization, the performance improved efficiently, particularly when these models were incorporated without early stopping.

Theoretical and Practical Implications

The findings challenge the use of Bayesian model averaging as a strategy for constructing performant deep ensembles, primarily due to its inherent inability to maintain diverse error patterns essential for ensemble effectiveness. In practical settings, deep learning practitioners could benefit from shifting focus toward simple yet well-weighted deep ensembles. The PAC-Bayesian framework stands out as a robust method to optimize these weights, ensuring both high generalization and formal performance guarantees.

Future Directions

Looking ahead, research could explore scaling PAC-Bayesian methods to larger and more complex datasets and architectures. Efficiently utilizing additional data for tandem loss estimation without sacrificing training data volume remains an open challenge. Incorporating PAC-Bayesian optimizations into end-to-end training pipelines, potentially automating the balance between individual model training and ensemble diversity, is another promising direction.

In sum, the paper by Hauptvogel and Igel offers a significant contribution to the understanding of ensemble methods in deep learning, clearly delineating the limitations of Bayesian ensembles and introducing PAC-Bayesian optimization as a compelling alternative for enhanced generalization performance.