2000 character limit reached
Promises and Pitfalls of the Linearized Laplace in Bayesian Optimization
Published 17 Apr 2023 in cs.LG and stat.ML | (2304.08309v2)
Abstract: The linearized-Laplace approximation (LLA) has been shown to be effective and efficient in constructing Bayesian neural networks. It is theoretically compelling since it can be seen as a Gaussian process posterior with the mean function given by the neural network's maximum-a-posteriori predictive function and the covariance function induced by the empirical neural tangent kernel. However, while its efficacy has been studied in large-scale tasks like image classification, it has not been studied in sequential decision-making problems like Bayesian optimization where Gaussian processes -- with simple mean functions and kernels such as the radial basis function -- are the de-facto surrogate models. In this work, we study the usefulness of the LLA in Bayesian optimization and highlight its strong performance and flexibility. However, we also present some pitfalls that might arise and a potential problem with the LLA when the search space is unbounded.
- Michael Aerni. On the Laplace approximation for sequential model selection of Bayesian neural networks. Master’s thesis, ETH Zurich, 2022.
- BoTorch: A framework for efficient Monte-Carlo Bayesian optimization. In NeurIPS, 2020.
- Bayesian experimental design for computed tomography with the linearised deep image prior. In ICML Workshop on Adaptive Experimental Design and Active Learning, 2022.
- Weight uncertainty in neural networks. In ICML, 2015.
- Conservative uncertainty estimation by fitting prior networks. In ICLR, 2020.
- Repulsive deep ensembles are Bayesian. In NeurIPS, 2021.
- On Stein variational neural network ensembles. arXiv preprint arXiv:2106.10760, 2021.
- Laplace redux–effortless Bayesian deep learning. In NeurIPS, 2021.
- Mixtures of Laplace approximations for improved post-hoc uncertainty in deep learning. In NeurIPS Workshop of Bayesian Deep Learning, 2021.
- Vincent Fortuin. Priors in Bayesian deep learning: A review. International Statistical Review, 90(3), 2022.
- BNNpriors: A library for Bayesian neural network inference with different prior distributions. Software Impacts, 9, 2021.
- Bayesian neural network priors revisited. In ICLR, 2022.
- Dropout as a Bayesian approximation: Representing model uncertainty in deep learning. In ICML, 2016.
- GPyTorch: Blackbox matrix-matrix Gaussian process inference with GPU acceleration. In NIPS, 2018.
- On upper-confidence bound policies for switching bandit problems. In ALT, 2011.
- Roman Garnett. Bayesian optimization. Cambridge University Press, 2023.
- Exact Langevin dynamics with stochastic gradients. arXiv preprint arXiv:2102.01691, 2021.
- John C. Gittins. Bandit processes and dynamic allocation indices. Journal of the royal statistical society series b-methodological, 41:148–164, 1979.
- Alex Graves. Practical variational inference for neural networks. In NIPS, 2011.
- Bayesian deep ensembles via the neural tangent kernel. In NeurIPS, 2020.
- Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem. In CVPR, 2019.
- Scalable marginal likelihood estimation for model selection in deep learning. In ICML, 2021a.
- Improving predictions of Bayesian neural nets via local linearization. In AISTATS, 2021b.
- Probing as quantifying inductive bias. In ACL, 2022a.
- Invariance learning in deep neural networks with differentiable Laplace approximations. In NeurIPS, 2022b.
- Averaging weights leads to wider optima and better generalization. In UAI, 2018.
- What are Bayesian neural network posteriors really like? In ICML, 2021.
- Neural tangent kernel: Convergence and generalization in neural networks. In NIPS, 2018.
- Efficient global optimization of expensive black-box functions. Journal of Global optimization, 13(4), 1998.
- Hands-on Bayesian neural networks - a tutorial for deep learning users. IEEE Computational Intelligence Magazine, 17(2), 2022.
- Fast and scalable Bayesian deep learning by weight-perturbation in Adam. In ICML, 2018.
- Approximate inference turns deep networks into Gaussian processes. In NeurIPS, 2019.
- Scalable and flexible deep Bayesian optimization with auxiliary information for scientific problems. TMLR, 2021.
- Adam: A method for stochastic optimization. In ICLR, 2015.
- Variational dropout and the local reparameterization trick. In Advances in Neural Information Processing Systems 28. 2015.
- Predictive uncertainty quantification with compound density networks. In NeurIPS Workshop in Bayesian Deep Learning, 2019.
- Being Bayesian, even just a bit, fixes overconfidence in ReLU networks. In ICML, 2020.
- Learnable uncertainty under Laplace approximations. In UAI, 2021a.
- An infinite-feature extension for Bayesian ReLU nets that fixes their asymptotic overconfidence. In NeurIPS, 2021b.
- Simple and scalable predictive uncertainty estimation using deep ensembles. In NIPS, 2017.
- Pierre-Simon Laplace. Mémoires de mathématique et de physique, tome sixieme. 1774.
- Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998.
- Deep neural networks as Gaussian processes. In ICLR, 2018.
- Structured and efficient variational deep learning with matrix Gaussian posteriors. In ICML, 2016.
- Multiplicative normalizing flows for variational Bayesian neural networks. In ICML, 2017.
- David JC MacKay. Bayesian interpolation. Neural computation, 4(3), 1992a.
- David JC MacKay. The evidence framework applied to classification networks. Neural Computation, 4(5), 1992b.
- David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3), 1992c.
- A simple baseline for Bayesian uncertainty in deep learning. In NeurIPS, 2019.
- James Martens. New insights and perspectives on the natural gradient method. JMLR, 21(146), 2020.
- Periodic activation functions induce stationarity. In NeurIPS, 2021.
- Laplacian autoencoders for learning stochastic representations. In NeurIPS, 2022.
- Data augmentation in Bayesian neural networks and the cold posterior effect. In UAI, 2022.
- Radford M Neal. Bayesian learning via stochastic dynamics. In NIPS, 1993.
- Radford M Neal et al. MCMC using Hamiltonian dynamics. Handbook of Markov Chain Monte Carlo, 2(11), 2011.
- Practical deep learning with Bayesian principles. In NeurIPS, 2019.
- The neural testbed: Evaluating predictive distributions. In NeurIPS, 2022.
- Carl Edward Rasmussen and Christopher K. I. Williams. Gaussian processes in machine learning. 2005.
- Pacoh: Bayes-optimal meta-learning with pac-guarantees. In ICML, 2021.
- PAC-Bayesian meta-learning: From theory to practice. arXiv preprint arXiv:2211.07206, 2022.
- Trust region policy optimization. In ICML, 2015.
- Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
- Last layer marginal likelihood for invariance learning. In AISTATS, 2022.
- Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1), 2015.
- Incorporating unlabelled data into Bayesian neural networks. arXiv preprint arXiv:2304.01762, 2023.
- Scalable Bayesian optimization using deep neural networks. In ICML, 2015.
- Sequential updating of conditional probabilities on directed graphical structures. Networks, 20(5), 1990.
- Bayesian optimization with robust Bayesian neural networks. In NIPS, 2016.
- Tycho FA van der Ouderaa and Mark van der Wilk. Learning invariant weights in neural networks. In UAI, 2022.
- High-dimensional Bayesian optimization with invariance. In ICML Workshop on Adaptive Experimental Design and Active Learning, 2022.
- Function space particle optimization for Bayesian neural networks. In ICLR, 2019.
- Bayesian learning via stochastic gradient langevin dynamics. In ICML, 2011.
- From predictions to decisions: The importance of joint predictive distributions. arXiv preprint arXiv:2107.09224, 2021.
- Bayesian deep learning and a probabilistic perspective of generalization. In NeurIPS, 2020.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.