Adaptive Delayed-Update Cyclic Algorithm for Variational Inequalities

Abstract: Cyclic block coordinate methods are a fundamental class of first-order algorithms, widely used in practice for their simplicity and strong empirical performance. Yet, their theoretical behavior remains challenging to explain, and setting their step sizes -- beyond classical coordinate descent for minimization -- typically requires careful tuning or line-search machinery. In this work, we develop $\texttt{ADUCA}$ (Adaptive Delayed-Update Cyclic Algorithm), a cyclic algorithm addressing a broad class of Minty variational inequalities with monotone Lipschitz operators. $\texttt{ADUCA}$ is parameter-free: it requires no global or block-wise Lipschitz constants and uses no per-epoch line search, except at initialization. A key feature of the algorithm is using operator information delayed by a full cycle, which makes the algorithm compatible with parallel and distributed implementations, and attractive due to weakened synchronization requirements across blocks. We prove that $\texttt{ADUCA}$ attains (near) optimal global oracle complexity as a function of target error $ε>0,$ scaling with $1/ε$ for monotone operators, or with $\log^2(1/ε)$ for operators that are strongly monotone.

• The paper introduces ADUCA which provides a parameter-free, locally adaptive step size mechanism for solving variational inequalities.
• The method leverages delayed cyclic updates, allowing independent block updates and efficient parallel computation without repeated line searches.
• Empirical results demonstrate near-optimal convergence rates in both monotone and strongly monotone settings across diverse datasets.

Adaptive Delayed-Update Cyclic Algorithm for Variational Inequalities

Introduction and Motivation

The paper "Adaptive Delayed-Update Cyclic Algorithm for Variational Inequalities" (2603.29128) addresses the development and analysis of cyclic block coordinate methods for general Minty variational inequalities (VIs). Classical cyclic block coordinate approaches are commonly used due to their simplicity and empirical efficiency, yet they are difficult to analyze rigorously. In VI settings, effective step size selection is a major challenge, particularly for methods that cannot rely on global or blockwise Lipschitz information or require extensive line search.

The core contribution is the ADUCA (Adaptive Delayed-Update Cyclic Algorithm), which is distinguished by its parameter-free, locally adaptive step size mechanism and a delayed-update structure allowing the use of stale operator information. This design is intended for broad classes of VIs, admitting monotone, locally (blockwise) Lipschitz operators without knowledge of corresponding constants. The approach enables effective parallelization and weak synchronization across variable blocks, making it suitable for scalable distributed implementations.

Algorithmic Framework

Formulation and Problem Setting

ADUCA targets generalized Minty VIs: Find $x^* \in \mathbb{R}^d$ such that for all $x \in \mathbb{R}^d$,

$$\langle F(x), x - x^* \rangle + g(x) - g(x^*) \ge 0,$$

with $F \colon \mathbb{R}^d \to \mathbb{R}^d$ monotone and locally blockwise Lipschitz, and $g$ convex, lower semicontinuous, and block-separable.

The method operates on partitioned variable blocks, implementing cyclic updates with operator information lagged by a full cycle. At initialization, a one-time line search is performed, but subsequent steps are fully parameter- and search-free.

Adaptive, Parameter-Free Step Sizes

A central innovation is the manner of stepsize selection. ADUCA estimates local Lipschitz constants using only information from recent iterates—eschewing any knowledge of global problem parameters. Specifically, per-cycle local Lipschitz surrogates, $L_k$ and $\hat{L}_k$, are empirically computed to modulate the stepsize $a_k$. The update uses the largest $a_k$ admissible according to simplified rules governed solely by the local geometry. The method also provides a robust initialization strategy to ensure convergence guarantees without repeated searches.

Notably, ADUCA requires no global Lipschitz constants, no per-epoch line search (post-initialization), and automatically adapts to the problem's local smoothness. This sets it apart from block and full-operator methods in the VI literature.

Delayed Update and Synchronization

A unique feature of ADUCA is its use of operator information lagging by one full coordinate cycle. This enables each block to update independently, with synchronization only needed across completed cycles, not individual block steps. Thus, the method is naturally aligned with modern distributed and parallel computing architectures, where synchronization bottlenecks are a key performance concern.

Convergence and Complexity Analysis

The paper establishes sharp ergodic convergence guarantees for ADUCA:

• For monotone operators, an $\mathcal{O}(1/\epsilon)$ oracle complexity (number of operator evaluations to reach $x \in \mathbb{R}^d$0-accuracy) is achieved, aligned with lower bounds for general monotone VIs [Nemirovski2004MirrorProx, ouyang2021lower].
• For strongly monotone cases (or restricted strong monotonicity), a near-linear rate in $x \in \mathbb{R}^d$1 is proven without requiring knowledge of the strong monotonicity parameter.

The proofs leverage new decomposition and control of telescoping error terms, careful tracking of merit functions, and analyze multi-step progress using a blend of primal-dual arguments and block-wise local Lipschitz surrogates.

Compared to prior work—such as CODER [SongDiakonikolas2023CODER], which requires global blockwise Lipschitz knowledge or per-epoch line search, or autoconditioned full-operator algorithms [malitsky2020golden, Alacaoglu2023BeyondGR]—ADUCA is the first to achieve these rates in a cyclic, parameter-free, and fully locally adaptive setting.

Numerical Evaluation

Comprehensive experiments evaluate ADUCA against state-of-the-art methods on convex elastic net-regularized support vector machine (SVM) problems over multiple datasets.

Figure 1 illustrates: the adaptivity of the ADUCA stepsize mechanism—showing that its local Lipschitz estimates ($x \in \mathbb{R}^d$2) are tighter and more informative than both global Lipschitz constants and surrogates used by competing methods.

Figure 1: Comparisons of global Lipschitz constants and local Lipschitz estimates used by different algorithms, on the Support Vector Machine problem with datasets a9a, gisette, and SUSY.

Key empirical findings:

• ADUCA consistently achieves the fastest, or comparable-to-best, convergence in primal suboptimality ($x \in \mathbb{R}^d$3) compared to parameter-free and parameter-dependent baselines, both with standard and diagonal-preconditioned Euclidean geometries.
• These results are robust across datasets with varying dimensions and block structures, demonstrating the method's ability to leverage block-adaptive geometry and cyclic updates without tuning.

Figure 2: Primal objective suboptimality $x \in \mathbb{R}^d$4 for the SVM problem, comparing ADUCA against alternatives on multiple real datasets.

Further sensitivity analysis (see Figure 3) confirms that the performance of ADUCA is insensitive to strong monotonicity parameter $x \in \mathbb{R}^d$5—even setting $x \in \mathbb{R}^d$6 yields indistinguishable efficiency, in line with theoretical expectations.

Figure 3: Sensitivity of ADUCA to the hyperparameter $x \in \mathbb{R}^d$7 used in the extrapolation weight $x \in \mathbb{R}^d$8; curves overlap, showing practical insensitivity.

Implications and Future Directions

Practical Impact

The delayed-update mechanism and adaptivity achieved by ADUCA directly address synchronization and hyperparameter bottlenecks in large-scale, distributed variational inequality and saddle-point problems. The one-shot initialization procedure removes the need for repeated costly line search, and all further adaptivity is achieved via local operator differences, thus making the approach naturally scalable.

In machine learning and distributed optimization, the algorithm is immediately applicable to large convex and monotone minimization, saddle-point problems, and games modeled as VIs with block structure. The algorithm's robustness against unknown problem parameters makes it especially relevant for auto-tuning and black-box scenarios common in real-world deployments.

Theoretical Significance

The analysis bridges gaps between block-cyclic and stochastic/randomized block update paradigms, establishing near-optimal complexity and convergence rates in a fully cyclic, parameter-free scheme. The design principles around delayed synchronization and local geometry estimation may generalize to other structured first-order algorithms.

Future refinements could focus on extending the framework to non-monotone and non-convex VI regimes, and on integrating variance reduction or acceleration techniques. Moreover, the theoretical question of linear rate guarantees when the monotonicity parameter is unknown, despite empirical evidence, remains open.

Conclusion

ADUCA constitutes a significant advance in cyclic block-coordinate optimization for variational inequalities. By enabling parameter-free, locally adaptive, and efficiently parallelizable updates via a delayed-informational mechanism, it sets a new standard for convergence and scalability in monotone VI solvers. Its robust theoretical and empirical characteristics make it a compelling methodology for both theoretical and applied researchers working in large-scale optimization, machine learning, and game-theoretic equilibrium computation.