Last-Layer Laplace Approximation

• Last-layer Laplace Approximation is a Bayesian inference technique that treats only the final neural network layer stochastically, yielding a tractable Gaussian posterior over its weights.
• It enables efficient uncertainty quantification, fast incremental learning, and improved active and conformal prediction through closed-form updates.
• The method leverages a second-order Taylor expansion around the MAP estimate and efficient Hessian inversion to bypass full-network retraining, trading off some fidelity for scalability.

The last-layer Laplace approximation is a methodology in approximate Bayesian inference where the Bayesian treatment is restricted solely to the final (linear) layer of a deep neural network, with all earlier layers held fixed at their maximum a posteriori (MAP) estimates. The technique yields a tractable Gaussian approximation for the posterior over the last-layer weights, enabling efficient uncertainty quantification and fast Bayesian updates—especially advantageous in scenarios where full network retraining is computationally prohibitive. It has become a key tool for scalable Bayesian deep learning, efficient incremental learning, active learning query strategies, and conformal prediction with calibrated uncertainty (Huseljic et al., 2022, Kim et al., 1 Dec 2025, McInerney et al., 2024).

1. Bayesian Setup and Motivation

Given network parameters $\omega \in \mathbb{R}^P$ and dataset $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$, the Bayesian posterior is $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$. For modern networks, $P$ is typically large, rendering posterior inference computationally intractable. The last-layer Laplace approach freezes the representation parameters $\theta$ (all layers except the last), and focuses on the posterior over the final weights $w \in \mathbb{R}^D$, often after a feature extraction step $\varphi(x) \in \mathbb{R}^D$. The marginal posterior then simplifies to $$p(w | \mathcal{D}, \theta) \propto p(w) \prod p(y_i | \varphi(x_i), w)$$, reducing parameter dimensionality from $P$ to $D$ (Huseljic et al., 2022, Kim et al., 1 Dec 2025, McInerney et al., 2024).

2. Derivation of the Gaussian Posterior (Laplace Approximation)

The last-layer Laplace approximation proceeds by expanding the log posterior density $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$0 to second order around its mode (MAP estimate) $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$1:

• MAP estimation: $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$2 With a Gaussian prior $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$3 and suitable likelihood (e.g., Gaussian for regression; logistic for binary classification).
• Hessian computation: Negative log-posterior Hessian at $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$4 is $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$5 For a regression model with homoscedastic noise $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$6, $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$7, where $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$8 is the design matrix (Kim et al., 1 Dec 2025). For logistic regression, $\mathcal{D} = \{(x_i, y_i)\}_{i=1}^N$9, with $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$0.
• Gaussian approximation: The posterior is approximated as $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$1, with $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$2 (Huseljic et al., 2022, Kim et al., 1 Dec 2025, McInerney et al., 2024).

3. Efficient Hessian Inversion and Fast Bayesian Updates

• Low-rank updates: Direct inversion of $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$3 costs $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$4. However, $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$5 is a low-rank modification of the prior precision, so the Sherman–Morrison–Woodbury identity or sequential rank-one updates permit inversion in $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$6 or $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$7 per update:
  • Initialize $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$8.
  • Sequentially update with each rank-one term:
  • $$p(\omega|\mathcal{D}) \propto p(\omega) \prod_{i=1}^N p(y_i|x_i, \omega)$$9 with $P$0.
  • For batch updates with new data, compute posterior update formulas for mean and covariance (Huseljic et al., 2022).
• Incremental updates: Given prior $P$1 and new data batch $P$2, perform a single Laplace step to yield new mean and covariance:
  • $P$3
  • Update $P$4 and $P$5 using the rank-one formulae; iterate as needed for numerical stability.
  • Prediction at $P$6 involves computing $P$7 (Huseljic et al., 2022).
• Computational advantage: Last-layer Laplace avoids retraining the full model or last layer entirely. Updates can be $P$8–$P$9 faster than gradient-based retraining for moderate $\theta$0 and $\theta$1; prediction is $\theta$2 versus multiple forward passes for MC methods (Huseljic et al., 2022).

4. Predictive Distributions and Uncertainty Decomposition

• Posterior predictive: For new $\theta$3 with feature map $\theta$4, the predictive distribution is

$\theta$5

for regression (Kim et al., 1 Dec 2025). The mean $\theta$6 is the standard point prediction; the variance decomposes into - $\theta$7: aleatoric (data) uncertainty, - $\theta$8: epistemic (model) uncertainty.

• Diagnostics: Fractional epistemic share $\theta$9 and overall posterior spread $w \in \mathbb{R}^D$0 are useful diagnostics for model uncertainty and interval tightness (Kim et al., 1 Dec 2025).

5. Applications: Incremental Learning, Active Learning, and Conformal Prediction

Application Area	Role of Last-Layer Laplace Approximation	Cited Paper
Incremental/Online Learning	Enables closed-form Bayesian updates on arrival of new data batches, faster than retraining	(Huseljic et al., 2022)
Active Learning	Permits fully sequential acquisition strategies (e.g., Uncertainty Sampling, BALD, Query-by-Committee) with per-point model update and uncertainty estimation	(Huseljic et al., 2022)
Conformal Prediction	Used in CLAPS to yield efficiency-improved conformal intervals with split-conformal calibration; intervals adapt to epistemic uncertainty	(Kim et al., 1 Dec 2025)
Uncertainty Quantification	Provides fast, scalable Gaussian posterior over outputs for downstream decision-making	(McInerney et al., 2024)

Active learning context: The method enables the use of computationally intensive acquisition functions (e.g., BALD) after each new label is acquired, rather than in large pre-selected batches. Empirically, SNGP-LA yields 2–5% better learning curves than traditional top-batch-based batch selection (Huseljic et al., 2022). Conformal prediction context: CLAPS combines the Gaussian posterior from last-layer Laplace with a two-sided posterior CDF conformity score, producing tighter prediction intervals with finite-sample coverage, particularly effective when epistemic variance is nontrivial (Kim et al., 1 Dec 2025).

6. Implementation Notes and Practical Considerations

• Cholesky and Hessian-vector products: For width $w \in \mathbb{R}^D$1 final-layer, Cholesky-based inversion is feasible unless $w \in \mathbb{R}^D$2 is extremely large. For large $w \in \mathbb{R}^D$3, conjugate-gradient (CG) methods can be used for Hessian solves (McInerney et al., 2024, Kim et al., 1 Dec 2025).
• Curvature approximations: In practice, the Gauss–Newton approximation is often used for the Hessian (McInerney et al., 2024). For further efficiency, diagonal, Kronecker-factored, or low-rank approximations can be applied.
• Numerical stability: Additional damping is applied to ensure positive-definiteness of the Hessian; fallback strategies are employed when the Hessian is ill-conditioned (McInerney et al., 2024).
• Scalability: The technique is computationally viable in settings where inverting the full-network Hessian is infeasible—applicable for last-layer dimensions up to several thousand.

7. Limitations and Theoretical Considerations

• Representation uncertainty ignored: Only last-layer weights are treated Bayesian. Uncertainty in the backbone is not captured; the method assumes feature extractor parameters are well-optimized and certain (Huseljic et al., 2022, Kim et al., 1 Dec 2025).
• Posterior contraction: As the data set size $w \in \mathbb{R}^D$4 increases, epistemic uncertainty diminishes ($w \in \mathbb{R}^D$5), so benefits over mean-based or residual-based methods vanish (Kim et al., 1 Dec 2025).
• Homogeneity assumptions: Many approaches assume homoscedastic Gaussian noise; strongly heteroscedastic settings require explicit modeling via scale or quantile heads (Kim et al., 1 Dec 2025).
• Fidelity vs. Scalability trade-off: High modeling fidelity would require full Laplace approximations over all parameters, which is computationally prohibitive. Last-layer Laplace trades off some fidelity for tractability (McInerney et al., 2024).
• Accuracy depends on feature representation: The posterior variance is only meaningful to the extent that the frozen representation $w \in \mathbb{R}^D$6 is expressive and robust (Kim et al., 1 Dec 2025).

• “Efficient Bayesian Updates for Deep Learning via Laplace Approximations” (Huseljic et al., 2022)
• “CLAPS: Posterior-Aware Conformal Intervals via Last-Layer Laplace” (Kim et al., 1 Dec 2025)
• “Variation Due to Regularization Tractably Recovers Bayesian Deep Learning” (McInerney et al., 2024)
• “An Extended Simplified Laplace strategy for Approximate Bayesian inference of Latent Gaussian Models using R-INLA” (Chiuchiolo et al., 2022)