Adaptive Bayesian Variable Selection in GLMMs

• Adaptive Bayesian variable selection in GLMMs is a rigorous framework that uses spike-and-slab priors and latent indicators for joint selection of fixed and random effects.
• It employs adaptive hyperparameter tuning and empirical Bayes updates integrated with advanced MCMC and variational algorithms to improve computational efficiency and accuracy.
• The approach is applicable to spatial and high-dimensional GLMMs, achieving notable performance in model recovery and error reduction across complex regression settings.

Adaptive Bayesian variable selection in generalized linear mixed models (GLMMs) constitutes a rigorous and flexible framework for simultaneously selecting fixed and random effects in complex regression settings. Leveraging spike-and-slab priors, latent indicator variables, and stochastic search or optimization procedures, adaptive approaches provide automatic control of model sparsity and accommodate uncertainty across the entire model space. Key developments include stochastic search variable selection (SSVS) with hierarchical hyperpriors, Bayesian adaptive Lasso via variational Bayes, adaptive MCMC proposals such as PARNI, and advanced mode-jumping MCMC schemes. This article details the model constructions, inference algorithms, adaptivity mechanisms, performance metrics, and practical recommendations underpinning modern adaptive Bayesian variable selection methodologies for GLMMs.

1. Model Specification and Prior Structures

GLMMs generalize standard regression by accommodating both fixed effects ($\beta$) and random effects ($u_i$) in the linear predictor $g(\mu_{ij}) = X_{ij}'\beta + Z_{ij}'u_i$ for observations $y_{ij}$, with conditional density governed by an exponential family. Adaptive Bayesian variable selection enriches this framework by assigning latent inclusion indicators to each fixed and random effect, enabling joint selection across all model terms.

In SSVS for GLMMs (Ding et al., 2024), fixed-effects coefficients $\beta_j$ are governed by

$$\beta_j \mid \gamma_j, \tau_0^2, \tau_1^2 \sim (1-\gamma_j)\,N(0,\tau_0^2) + \gamma_j\,N(0,\tau_1^2), \qquad \gamma_j \sim \mathrm{Bernoulli}(\pi_\beta),$$

where $\tau_0^2$ is a strong shrinkage ("spike") and $\tau_1^2$ allows inclusion ("slab"). For random effects, spikes are induced via the modified Cholesky decomposition $\Omega = \Lambda\Gamma\Gamma'\Lambda$; $\lambda_k=0$ switches off the $k^\text{th}$ random term. Latent indicators $\delta_k$ control inclusion: $$\lambda_k \mid \delta_k, \tau_\lambda^2 \sim \delta_k\,N_+(0, \tau_\lambda^2 h^2) + (1-\delta_k)\,\delta_0(\lambda_k),$$ with $\delta_k \sim \mathrm{Bernoulli}(\pi_u)$ and $N_+(\cdot)$ truncated positive.

Alternative frameworks, such as Bayesian adaptive Lasso (Tung et al., 2016), impose double-exponential (Laplace) priors on fixed effects, with shrinkage hyperparameters $\lambda_j \sim \mathrm{Gamma}(r,s)$. Each $\beta_j \mid \lambda_j \sim \mathrm{DE}(\lambda_j)$, offering continuous adaptivity to coefficient magnitude.

Approaches for spatially dependent GLMMs extend the indicator concept to area-specific inclusion via latent probit thresholding, with indicators $\gamma_{ij}=1_{z_{ij}>0}$ for region $i$ and covariate $j$, and conditionally autoregressive (CAR) priors over the latent $z_j$ vector (Lum, 2012).

Group selection in structured additive mixed models (GAMM/GLMM) employs parameter-expanded spike-and-slab hierarchies on coefficient groups, introducing scalar importance parameters and group-level inclusion indicators (Scheipl, 2011).

2. Hyperparameter Adaptation and Empirical Bayes Tuning

Crucial to adaptive variable selection in GLMMs is the dynamic adjustment of hyperparameters governing inclusion probabilities and variance components. Standard practice is to endow inclusion rates ($\pi_\beta$, $\pi_u$) and slab/spike variances ($\tau_1^2$, $h^2$) with hierarchical priors (commonly Beta for probabilities and inverse-Gamma for variances). Empirical Bayes updating is incorporated by computing within-chain posterior means of indicators (e.g., $\hat{\pi}_\beta = E[\gamma_j]$) and plugging these directly into the prior distribution.

In SSVS (Ding et al., 2024), grid search preliminary chains over $(h,v)$ permit localization of stable operating regions for the hyperparameters, e.g., $h \in \{0.1,1,5\}$, $v\in \{0.1,1,10\}$. Within MCMC, adaptation of Metropolis-Hastings proposal scales for Cholesky parameters ($\Gamma$) is performed to target desired acceptance rates.

For spikeSlabGAM (Scheipl, 2011), hyperparameters such as slab variances $a_\tau, b_\tau$, spike magnitude $v_0$, and global inclusion probability $w$ are given default priors shown to perform robustly in extensive empirical settings. These adapt via block Gibbs steps and are monitored via model size and inclusion probability statistics.

In the Bayesian adaptive Lasso (Tung et al., 2016), hyperparameters $\lambda_j$ self-tune through variational equation updates, enforcing heavier shrinkage for coefficients near zero and relaxing for large signals, embodying "automatic relevance determination".

3. Computational Algorithms for Inference

Adaptive Bayesian variable selection in GLMMs typically relies on tailored MCMC or variational Bayes routines:

• SSVS MCMC (Ding et al., 2024): Alternately updates $\beta$, $\gamma$, $\lambda$, $\delta$, $\Gamma$, $\xi$, and all hyperparameters. Gibbs steps update inclusion indicators and variance components, while Metropolis-Hastings steps control non-conjugate parameters. Data augmentation (Polya-Gamma for logistic, Laplace for other non-Gaussian likelihoods) facilitates conditionally normal updates.
• Adaptive Lasso Variational Bayes (Tung et al., 2016): Employs coordinate ascent, cycles between updates for $\beta$ (posterior mode under shrinkage), $b$ (random effects via local Gaussian Laplace approximation), shrinkage parameters $\lambda_j$, and random-effect precision $Q$. No Laplace integration over the likelihood is required.
• Spatial SSIP MCMC (Lum, 2012): Combines Gaussian latent variable sampling for $z_{ij}$ (CAR-prior), conjugate updates for slab variances, and Polya-gamma or Albert-Chib augmentation for non-Gaussian GLMs. Block Gibbs sampling over multivariate components and efficient linear system solvers scale to large $n,p$.
• PARNI Adaptive MCMC (Liang et al., 2023): Constructs locally informed random-neighbourhood MCMC proposals, with adaptive jump rates ($A_j$, $D_j$) tuned by running estimates of posterior inclusion probabilities. Marginal likelihoods are approximated via Laplace or approximate Laplace (ALA), with warm-start initial guesses and Newton-Raphson refinement, greatly reducing per-iteration computational costs.
• Mode-jumping MCMC (MJMCMC) (Hubin et al., 2016): Augments local single-flip MH moves with occasional large-jump proposals consisting of index swaps, local optimization (simulated annealing, greedy, or multiple-try MCMC), and random perturbations. Acceptance is computed via auxiliary-variable MH, and multiple-try options enhance exploration in high-dimensional multimodal model spaces.

4. Evaluation: Simulation Settings, Metrics, and Performance

Recent simulation studies (Ding et al., 2024) characterize accuracy and computational efficiency using root-mean-square error (RMSE) for coefficient estimation, true-model recovery rates, and CPU utilization. For instance, SSVS with diagonal $\Gamma$ achieves 44% recovery in the sparse case ($l=q=10, n=100$), RMSE for $\beta$ at $5 \times 10^{-4}$ versus $5.7 \times 10^{-4}$ for a basic GLMM, with computational cost scaling favorably as $q$ increases.

For Bayesian adaptive Lasso (Tung et al., 2016), experiments demonstrate higher Correct-Fitted-Rates (CFR) on zero vs nonzero selection, with VBGLMM outperforming penalized-Laplace GLMMLasso both in accuracy and speed (e.g., 44s vs 2230s per run).

PARNI adaptive MCMC (Liang et al., 2023) yields $3$–$10\times$ lower mean-squared error on posterior inclusion probabilities per CPU time, with scalability shown on large-scale gene-mapping data ($p$ up to $40{,}000$).

For spatial SSIP (Lum, 2012), simulation studies and real-data analysis demonstrate model recovery and parameter inference benefits in regionally varying GLMs. Performance metrics include marginal posterior inclusion probability and predictive distribution calibration.

5. Practical Considerations and Implementation Guidance

Recommended practice for adaptive Bayesian variable selection includes:

• Prior Choices: Uniform Beta(1,1) priors for inclusion probabilities unless substantive prior information is available. For spike-and-slab, select a small spike variance ($\tau_0^2 \approx 10^{-4}$), a large slab variance ($\tau_1^2$ in $[1,10]$), or validated group prior parameters for coefficient blocks.
• Hyperparameter Tuning: Conduct short pilot chains or grid searches to assess stability (monitor inclusion probabilities and mixing), and adjust slab-to-spike ratios and proposal scales. Use empirical-Bayes plug-in estimates where feasible.
• Initialization: Fit standard GLMMs (e.g., via glmmTMB, brms) for starting values of $\beta$ and $\Omega$; initialize all indicators to inclusion.
• MCMC Diagnostics: Monitor model size ($\sum_j \gamma_j$), random-effect count ($\sum_k \delta_k$), and mixing of key parameters. Track posterior inclusion probabilities—values $>0.5$ or $>0.9$ indicate robust inclusion.
• Posterior Analysis: Select terms with marginal posterior inclusion probability above a threshold (e.g., 0.5 for the median probability model), and perform Bayesian model averaging or refit reduced models without shrinkage for predictive checking.
• Computational Tools: Accessible implementation in JAGS (runjags), Stan/brms (with custom mixtures), glmmvb (VB-Lasso in R), and tailored MCMC packages as warranted by model structure.

6. Comparative Methodologies and Recent Extensions

The landscape of adaptive Bayesian variable selection in GLMMs includes several prominent strategies:

• SSVS and Related Spike-and-Slab Models: Provides precise probabilistic model selection for fixed and random effects via stochastic indicators and hierarchical hyperpriors (Ding et al., 2024, Scheipl, 2011).
• Bayesian Adaptive Lasso: Continuous shrinkage allows self-tuning via variational updates, competitive in high dimensions with straightforward implementation (Tung et al., 2016).
• Spatially Structured Selection: For areal or lattice-based GLMMs, latent probit models enforce spatial dependence among inclusion probabilities (Lum, 2012).
• Adaptive MCMC Proposals: PARNI and MJMCMC improve sampling efficiency and space exploration, with pointwise and large-jump mechanisms specifically engineered for multimodal posteriors (Liang et al., 2023, Hubin et al., 2016).
• Laplace and Quasi-Likelihood Integration: Laplace approximation or ALA enable tractable marginal likelihood computation for model evidence in non-Gaussian mixed models (Ding et al., 2024, Liang et al., 2023, Hubin et al., 2016).

The field continues to evolve, with ongoing work focused on scalability to high-dimensional $p$, robustness to prior/hyperparameter choices, extensions to group and hierarchical sparsity, and computational parallelization.

7. Summary and Outlook

Adaptive Bayesian variable selection in GLMMs combines hierarchical prior specification, latent indicators, and algorithmic tuning for robust model identification in complex mixed-effects regressions. SSVS approaches, adaptive Lasso, spatially structured priors, and informed MCMC proposals offer a wide spectrum of solutions scalable to large $p$, multi-level random effects, and general exponential family likelihoods. Empirical benchmarks consistently show improvements in accuracy, computational cost, and posterior inference over penalized or frequentist methods. Continuing advances will likely address higher-order interaction selection, ultra-high-dimensional predictors, and integration with flexible prior architectures for compositional and nonparametric effects.

Key References:

• Stochastic Search Variable Selection for Bayesian Generalized Linear Mixed Effect Models (Ding et al., 2024)
• Bayesian Adaptive Lasso with Variational Bayes for Variable Selection in High-dimensional GLMMs (Tung et al., 2016)
• Adaptive MCMC for Bayesian variable selection in generalised linear models and survival models (Liang et al., 2023)
• Bayesian variable selection for spatially dependent generalized linear models (Lum, 2012)
• spikeSlabGAM: Bayesian Variable Selection, Model Choice and Regularization for Generalized Additive Mixed Models in R (Scheipl, 2011)
• Mode jumping MCMC for Bayesian variable selection in GLMM (Hubin et al., 2016)