Monte Carlo Structured SVI (MC-SSVI)

• MC-SSVI is an advanced variational inference framework that integrates structured variational approximations with Monte Carlo estimation to provide tighter evidence bounds in non-conjugate models.
• It combines stochastic natural-gradient updates with mini-batching to enable scalable and efficient Bayesian inference in hierarchical latent variable models.
• The method has been successfully applied to mixed-effects models, sparse Gaussian processes, and probabilistic matrix factorization, yielding improved convergence and predictive performance.

Monte Carlo Structured Stochastic Variational Inference (MC-SSVI) is an advanced variational inference framework for scalable Bayesian inference in hierarchical latent variable models, with particular emphasis on two-level models that do not require conjugacy. It generalizes the SVI paradigm by allowing structured variational families and Monte Carlo estimation of intractable expectations, enabling effective learning in non-conjugate models by combining stochastic natural-gradient optimization, mini-batching, and flexible variational dependence structures. MC-SSVI has been applied successfully to mixed-effects models, sparse Gaussian processes, probabilistic matrix factorization, and correlated topic models, yielding improved statistical fidelity and convergence behavior over prior mean-field SVI and black-box variational methods (Sheth et al., 2016, Hoffman et al., 2014).

1. Structured Variational Approximations

Traditional SVI relies on mean-field variational families, positing full independence among global and local latent variables (e.g., $q(w,f) = \prod_j q(w_j) \prod_i q(f_i)$). This independence, however, limits expressivity and creates suboptimal, loose evidence bounds. MC-SSVI expands the variational family along a spectrum of structured dependency:

• Mean-field: $q(w,f) = \prod_j q(w_j) \prod_i q(f_i)$; full global-local independence.
• Simple structured: $q(w,f) = q(w) p(f|w)$; introduces some dependencies but cannot adapt $p(f|w)$.
• True structured: $q(w,f) = q(w) q(f|w)$ where $q(f|w)$ is a free variational factor. Setting $q(f|w) = p(f|w,y)$ yields the optimal structured bound and always tightens the Evidence Lower Bound (ELBO) over mean-field alternatives.

In this setup, the structured ELBO becomes:

$$L[q] = -\mathrm{KL}(q(w)\Vert p(w)) + \mathbb{E}_{q(w)}[\log p(y|w)]$$

where $$p(y|w) = \mathbb{E}_{p(f|w)}\left[\prod_i p(y_i|f_i)\right]$$. When the model's structure allows, this further decomposes as:

$$L = -\mathrm{KL}(q(w)\Vert p(w)) + \sum_i \mathbb{E}_{q(f_i)}[\log p(y_i|f_i)]$$

enabling scalable, mini-batch computation essential for large datasets (Sheth et al., 2016, Hoffman et al., 2014).

2. MC-SSVI Methodology: Gradients and Optimization

MC-SSVI optimizes the (typically intractable) structured ELBO using both natural-gradient and ordinary-gradient updates. Assuming $p(w)$, $q(w)$ are in the same exponential family,

$$p(w) = \exp\left(t(w)^T\theta_p - F(\theta_p)\right) h(w),\quad q(w) = \exp\left(t(w)^T\theta_q - F(\theta_q)\right) h(w)$$

with expectation parameters $\eta_q = \mathbb{E}_q[t(w)]$:

• Natural-gradient fixed-point: $\theta_q \leftarrow \theta_p + G(\eta_q)$ with $G(\eta_q) = \partial L/\partial \eta_q$.
• Stochastic natural-gradient: For mini-batch $M$ of size $|M|$,

$$\hat{G}(\eta_q) \approx \frac{N}{|M|} \sum_{i\in M} G_i(\eta_q),\quad \theta_q \leftarrow (1-\rho)\theta_q + \rho\left(\theta_p + \hat{G}(\eta_q)\right)$$

• Ordinary-gradient: For standard parameters $\phi_q$,

$$\phi_q \leftarrow \phi_q + \rho \partial L/\partial \phi_q$$

These updates are efficiently estimated using Monte Carlo methods suitable for the model structure (Sheth et al., 2016).

3. Monte Carlo Estimation of Intractable Expectations

Key expectations—specifically, $\mathbb{E}_{q(f|w)}[\log p(y_i|f_i)]$—are usually intractable in non-conjugate settings. MC-SSVI employs analytic identities and Monte Carlo samples:

• Latent Gaussian case (GLM, PMF): For $q(f_i) = \mathcal{N}(m_i, v_i)$,

$$\gamma_i = \frac{\partial}{\partial v_i} \mathbb{E}_{\mathcal{N}(f_i|m_i,v_i)}[\log p(y_i|f_i)] = \frac{1}{2} \mathbb{E}_{\mathcal{N}(f_i|m_i,v_i)}\left[\frac{\partial^2}{\partial f_i^2}\log p(y_i|f_i)\right]$$

Approximated by averaging the second derivatives over sampled $f_i$.

• Probabilistic Matrix Factorization: For each $(i,j)$, sample $u_i^a \sim q(u_i)$, $$f_{ij}^{a,b} \sim \mathcal{N}(m_{v_j}^T u_i^a, (u_i^a)^T S_{v_j} u_i^a)$$, then aggregate the Hessian terms.
• Correlated Topic Models: For $\xi(\eta_d)$,

$$\frac{\partial}{\partial S_{ij}}\mathbb{E}_{q(\eta)}[\xi] = \frac{1}{2}\mathbb{E}_q\left[\frac{\partial^2\xi}{\partial \eta_i \partial \eta_j}\right]$$

Samples of $\eta_d^{(\ell)}$ are drawn to estimate the second moments.

This approach exploits analytic structure while requiring no additional variance-reduction methods (Sheth et al., 2016).

4. Hybrid Natural and Standard Gradient Updates

In models where global variables are latent Gaussians (LGM), empirical evidence shows contrasting behaviors for different parameter updates:

• Covariance ($S$): Natural-gradient updates yield rapid, stable convergence.
• Mean ($m$): Natural-gradient updates may cause oscillation or learning instability.

The hybrid MC-SSVI (H-MC-SSVI) algorithm addresses this by updating the covariance parameter via stochastic natural-gradient, while the mean parameter is updated via an ordinary-gradient. For parameter vectors $(\theta_\mathrm{mean}, \theta_\mathrm{cov})$:

$$\theta_\mathrm{cov} \leftarrow (1-\rho)\theta_\mathrm{cov} + \rho(\theta_{p,\mathrm{cov}} + \hat{G}_\mathrm{cov}(\eta)),$$

while $\theta_\mathrm{mean}$ is updated with the standard gradient (Sheth et al., 2016). This hybridization yields both rapid convergence and stability in practice.

5. Comparison to Prior SVI Methodologies

MC-SSVI introduces several advancements over prior SVI, mean-field, and black-box variational frameworks:

• Non-conjugate Support: Unlike Hoffman et al. (2013, 2015) (Hoffman et al., 2014), MC-SSVI applies to non-conjugate models so long as $q(w)$ matches the prior's exponential-family, enabling natural gradients.
• Optimal Structured Bound: By employing the optimal $q(f|w)$ factor, MC-SSVI achieves systematically tighter ELBOs than methods restricted to mean-field or fixed conditional forms.
• Empirical Efficiency: MC-SSVI attains improved convergence speeds and robustness to step-size changes over black-box DSVI (Titsias 2014) and reparameterization-based methods (Kingma & Welling 2014; Rezende et al. 2014), due to natural-gradient-based updates (Sheth et al., 2016).
• Scalability: The decomposition of the ELBO into per-datapoint terms enables mini-batch (stochastic) updates, yielding scalability to large $N$.

A summary table:

Property	Mean-field SVI	Black-box VI	MC-SSVI (structured)
Non-conjugacy allowed	X	✔	✔
Structured dependency	X	Partial	Full (optimal possible)
Mini-batch enabled	✔	✔	✔
Natural-gradient support	Partial	X	✔

6. Applications and Empirical Evaluations

MC-SSVI and its hybrid form H-MC-SSVI have been applied and evaluated on a broad suite of models:

• Generalized Mixed-Effects GLM: On models with Gaussian weights and Rayleigh noise, H-MC-SSVI achieved lower test negative log-likelihood and faster convergence than mean-field or S-DSVI alternatives.
• Sparse Gaussian Processes: Variational bounds optimized by MC-SSVI are tighter (ELBO closer to the optimum) than those attainable using the standard Titsias (2009) bound. Approximate variants V₁ and V₂ retain favorable computational scaling ($\mathcal{O}(M^3 + NM^2)$).
• Probabilistic Matrix Factorization: On both synthetic and real datasets (binary, count, ordinal, continuous), H-MC-SSVI achieves faster and more stable ELBO convergence, with lower test NLL and error rates compared to S-DSVI.
• Correlated Topic Models: H-MC-SSVI supports a larger number of topics without overfitting and produces higher ELBO and lower test NLL than mean-field or simple structured methods.

In all cases, MC-SSVI's improved dependency modeling and efficient Monte Carlo/natural-gradient combination significantly enhance inference quality and convergence (Sheth et al., 2016).

7. Algorithmic Workflow and Convergence

The MC-SSVI-A algorithm is formalized as follows (Hoffman et al., 2014):

1. Draw global variable sample $\theta$ from $q(\theta; \lambda)$ by the quantile transform.
2. For each data group $n$ (or mini-batch $S$):
   • Update local factors $\gamma_n$ by maximizing the local ELBO,
   • Draw $M$ samples $z_n^{(m)}$ from $q(z_n|\theta; \gamma_n)$,
   • Estimate local sufficient statistics $$\hat{\eta}_n = \frac{1}{M}\sum_{m=1}^M \eta_n(x_n, z_n^{(m)})$$.
3. Compute stochastic (natural) gradient $$g_A = -\lambda + \eta + (N/|S|)\sum_{n\in S} \hat{\eta}_n$$.
4. Update $\lambda$ using a Robbins–Monro step.

Convergence to a stationary point of the ELBO follows under standard Robbins–Monro step-size conditions. Empirical studies demonstrate MC-SSVI achieves lower predictive errors and greater robustness to hyperparameters than mean-field SVI, with improved local minima avoidance (Hoffman et al., 2014). For example, in large-scale LDA, MC-SSVI reduces predictive log likelihoods by 10–20% over mean-field, and in Dirichlet-process mixtures, more accurately recovers the true number of components.

---

MC-SSVI systematically broadens the applicability and empirical power of variational inference for complex, large-scale Bayesian models beyond the restrictive boundaries of mean-field and conjugacy requirements, leveraging structured variational dependencies and stochastic natural-gradient learning to advance the state of scalable Bayesian inference (Sheth et al., 2016, Hoffman et al., 2014).