Stochastic Variance-Reduced FBHF Splitting

• The paper introduces a stochastic variance-reduced algorithm that generalizes FBHF splitting, offering strong convergence guarantees under flexible structural assumptions.
• It leverages the finite-sum structure by using stochastic estimators and reference updates to reduce per-iteration complexity from O(N) to constant time.
• Empirical results demonstrate linear convergence and significant reductions in CPU time compared to deterministic methods in large-scale optimization.

The stochastic variance-reduced forward-backward-half forward splitting (VRFBHF) algorithm is a state-of-the-art operator splitting scheme for solving structured monotone inclusion problems in Hilbert spaces. It addresses inclusions where the sum consists of a maximally monotone operator, a maximally monotone and Lipschitz continuous operator (typically with finite-sum structure), and a cocoercive operator. By integrating variance-reduced stochastic updates within the forward-backward-half forward (FBHF) framework, VRFBHF generalizes and improves upon classic deterministic and stochastic splitting methods, providing strong convergence guarantees under flexible structural assumptions while achieving substantial computational benefits in large-scale settings.

1. Structured Monotone Inclusions and Algorithmic Framework

The prototypical problem solved by VRFBHF is to find $x \in \mathbb{R}^d$ satisfying

$0 \in (A+B+C)(x),$

with

• $A: \mathbb{R}^d \rightrightarrows \mathbb{R}^d$ maximally monotone (possibly set-valued),
• $B: \mathbb{R}^d \to \mathbb{R}^d$ single-valued, $L_B$-Lipschitz, maximally monotone, and of finite-sum form $B = \sum_{i=1}^N B_i$, each $B_i$ $L_i$-Lipschitz,
• $C: \mathbb{R}^d \to \mathbb{R}^d$ $\beta$-cocoercive.

This structure encompasses a diverse range of applications, including constrained convex optimization, finite-sum composite minimization, and structured variational inequalities where constraints and penalties naturally admit a splitting.

The stochastic VRFBHF algorithm maintains two coupled sequences $\{x^k\}$ and $\{w^k\}$ and employs stochastic variance-reduced estimators to access $B$ without computing the full sum at each iteration: $$\begin{aligned} \bar{x}^k &= \lambda x^k + (1-\lambda) w^k \ y^k &= J_{\gamma A} \left( \bar{x}^k - \gamma (B+C)w^k \right) \ x^{k+1} &= y^k + \gamma \left( B_{\xi_k} w^k - B_{\xi_k} y^k \right) \ w^{k+1} &= \begin{cases} x^{k+1}, & \text{ w.p. } p \ w^k, & \text{ w.p. } 1-p \end{cases} \end{aligned}$$ $J_{\gamma A}$ is the resolvent: $J_{\gamma A} = (I + \gamma A)^{-1}$. $B_{\xi_k}$ denotes a stochastic estimator of $B$ constructed via either uniform or importance sampling, and $\lambda \in (0,1)$ tunes the extrapolation.

2. Variance Reduction Mechanism and Stochastic Iterates

Variance reduction is achieved by replacing $B$ with $B_{\xi_k}$ that leverages the finite-sum structure of $B$. At each iteration, a coordinate or mini-batch is sampled (distribution $Q$), and only the selected components $B_i$ are computed. This reduces per-iteration complexity from $O(N)$ to potentially constant time, a key advantage for large $N$.

The sequence $w^k$ acts as a stochastically updated reference point (anchor), updated to $x^{k+1}$ with probability $p$, otherwise kept fixed. This mechanism, akin to "loopless" SVRG, allows VRFBHF to maintain unbiased estimates and adapt variance dynamically, while extrapolation via $\lambda$ improves stability and convergence.

3. Convergence Analysis and Lyapunov Framework

Convergence guarantees for VRFBHF are established under mild operator assumptions:

• $A$ maximally monotone,
• $B$ monotone, single-valued, $L_B$-Lipschitz with unbiased stochastic estimator $B_\xi$ of controlled variance,
• $C$ $\beta$-cocoercive,
• Existence of a solution: $\mathrm{zer}(A+B+C)$ nonempty.

A Lyapunov function is introduced to measure the combined progress of current and reference iterates: $$\Phi_k(x^*) := \lambda \| x^k - x^* \|^2 + \frac{1-\lambda}{p} \| w^k - x^* \|^2$$ for any fixed solution $x^*$. The expected decrease property

$\mathbb{E}_k[ \Phi_{k+1}(x^*) ] \leq \Phi_k(x^*)$

ensures that the sequence $\{x^k\}$ converges almost surely (weakly) to a solution. This supermartingale property forms the backbone of the convergence proof.

When either $A$ or $B$ is strongly monotone (with modulus $\mu>0$), a contraction in expectation is established: $$\mathbb{E}\|x^k-x^*\|^2 \leq \left(\tfrac{1}{1+c/4}\right)^k \cdot \frac{2}{1-p} \|x^0 - x^*\|^2,$$ for an explicit $c>0$ depending on monotonicity and algorithmic parameters, implying exponential decay (linear rate) in mean squared error.

4. Numerical Performance and Empirical Observations

Comprehensive experiments on constrained convex optimization (monotone inclusions with linear/nonlinear constraints) empirically validate VRFBHF. Compared to deterministic FBHF-type algorithms, VRFBHF achieves:

• Significantly reduced computation time and number of iterations, especially as $N$ or $d$ increases,
• Equal or superior error decay (measured via duality gap or distance to solution),
• Robustness to the selection of reference update probability $p$; optimal convergence is empirically achieved for moderate $p$ (e.g., $p=0.2$).

Empirical results confirm that stochastic updates maintain convergence while dramatically lowering per-iteration cost, a critical feature for large-scale optimization.

Problem Size ($N$)	Deterministic FBHF: CPU/Iterations	VRFBHF: CPU/Iterations
Small	Higher	Lower
Large	Substantially higher	Significantly lower

(Figures/tables in the original work substantiate these comparisons.)

5. Generalization and Connections to Other Splitting Methods

VRFBHF subsumes classical forward-backward and forward-backward-half forward methods and extends the applicability of stochastic splitting beyond settings requiring all operators to have finite-sum structures. The algorithmic template allows variance reduction without averaging, preserving properties such as sparsity of iterates, and retains strong theoretical guarantees for stochastic operator splitting. VRFBHF also improves upon variance-reduced extragradient, stochastic FBF, and other splitting approaches by marrying lower complexity with general convergence theory (almost sure and linear rates).

The Lyapunov-based analysis, reference point randomization, and handling of finite-sum/coordinate structure are shared motifs with recently developed accelerated and momentum-based splitting schemes, but VRFBHF provides a concretely implementable, parameter-robust, and highly scalable alternative.

6. Implications and Applicability

VRFBHF enables practical solution of large-scale monotone inclusions and variational inequalities in settings where the cost of summing all operator components is prohibitive. It offers a unifying principle for stochastic and variance-reduced splitting algorithms, with theoretical assurances of convergence and real-world efficacy supported by empirical performance, and is particularly effective in machine learning and signal processing applications with structured regularization or complex constraints.

The approach is broadly compatible with recent trends in operator splitting—including nonlinear kernel corrections and momentum—but demonstrates that much of the practical benefit can be achieved with a simple stochastic, variance-reduced randomization mechanism, without sacrificing provable convergence.