Randomized Feasibility Algorithm with Polyak Steps

• The paper introduces a randomized feasibility algorithm that replaces full projection onto intersected constraints with tractable, sampled Polyak subgradient updates.
• It employs adaptive, parameter-free step-size strategies to achieve linear convergence in strongly convex cases and optimal sublinear rates for general convex functions.
• Empirical evaluations on QCQP and SVM tasks demonstrate that the method maintains computational efficiency and competitive performance without rigorous parameter tuning.

A randomized feasibility algorithm with Polyak steps is a class of iterative methods for constrained convex optimization where computationally tractable projections onto each individual constraint set are used instead of direct projection onto the intersection of all constraints. At each iteration, the algorithm randomly samples constraints and projects the current point towards feasibility using subgradient steps of Polyak type. Adaptive, problem-parameter-free step-size rules and sampled constraint selection enable linear or sublinear convergence rates according to the regularity of the objective function, while maintaining computational practicality when the full constraint projection is prohibitive (Chakraborty et al., 27 Jan 2026).

1. Problem Formulation and Notation

The central problem is of the form

$$\text{minimize} \quad f(x) \quad \text{subject to} \quad x \in X \cap Y,$$

where

• $f: \mathbb{R}^n \to \mathbb{R}$ is convex (possibly strongly convex and/or smooth),
• %%%%1%%%%, with each $g_i$ convex,
• $Y \subset \mathbb{R}^n$ is a simple closed convex set (such as a box or Euclidean ball).

Key notations:

• $\|\cdot\|$ is the Euclidean norm,
• $\Pi_Y[z]$ denotes projection onto $Y$,
• $g^+(x) := \max\{0, g(x)\}$,
• $$\text{dist}(x, X \cap Y) := \min_{y \in X \cap Y} \|x - y\|$$.

A global error bound assumption is used: there exists $c > 0$ and a sampling distribution $\omega$ over $\{1,\ldots,m\}$ such that for all $x \in Y$,

$$\text{dist}^2(x, X \cap Y) \le c\,\mathbb{E}_\omega\left[ (g_\omega^+(x))^2 \right].$$

2. Randomized Feasibility Algorithm with Polyak Steps

The algorithm performs a sequence of feasibility updates, each consisting of $N_k$ substeps at iteration $k$. Each feasibility substep involves:

• Sampling a constraint index $\omega \in \{1,\ldots,m\}$ uniformly,
• Computing a subgradient $d^i \in \partial g_\omega^+(z^{i-1})$,
• Updating via the Polyak-type step: $$z^i = \Pi_Y\left[ z^{i-1} - \beta \frac{g_\omega^+(z^{i-1})}{\|d^i\|^2} d^i \right],$$ where $\beta \in (0,2)$ is a parameter, and projection is onto $Y$.

After $N_k$ such substeps, $x_k = z^{N_k}$. This scheme avoids projection onto $X \cap Y$, replacing it with computationally tractable projections onto $Y$ and randomized selection of individual constraints.

Under the error-bound and bounded subgradient assumptions, the following hold:

• Nonexpansiveness: For any feasible $x \in X \cap Y$, $\|x_k - x\| \le \|v_k - x\|$.
• Geometric decrease in infeasibility: $$\mathbb{E}\left[ \text{dist}^2(x_k, X \cap Y) \mid v_k \right] \le (1 - q)^{N_k} \text{dist}^2(v_k, X \cap Y),$$ where $q = \beta(2-\beta)/(c M_g^2)$.

3. Interleaved Objective Minimization and Feasibility Updates

The algorithm alternates or interleaves randomized feasibility updates with (sub)gradient steps for objective minimization. Two major cases are considered:

Strongly Convex, $L$-Smooth Objective

Assumptions:

• $f$ has $L$-Lipschitz gradient,
• $f$ is $\mu$-strongly convex.

Algorithm steps:

1. Compute $v_{k+1} = \Pi_Y[x_k - \alpha_k \nabla f(x_k)]$,
2. Update $x_{k+1}$ using the randomized feasibility algorithm with $v_{k+1}, N_{k+1}$.

Adaptive Polyak-type step size: $$\alpha_k = \min\left\{ \frac{1}{2(L-\mu)}, \frac{1}{L}, \frac{\epsilon}{2\|\nabla f(x_k)\|^2} \right\},$$ where $\epsilon$ is a prescribed accuracy.

Weighted averaging is used: $$\bar{x}_k = \frac{\sum_{t=1}^k (1 - \bar{\mu})^{k-t} \alpha_t x_t}{\sum_{s=1}^k (1 - \bar{\mu})^{k-s} \alpha_s}, \quad \bar{\mu} = \min\left\{ \frac{1}{2(L-\mu)}, \frac{1}{L}, \frac{\epsilon}{2 M_f^2} \right\}.$$

Convex, Possibly Nonsmooth Objective: Distance-over-Weighted-Subgradients (DoWS)

Assumptions:

• $f$ is convex (possibly nondifferentiable),
• $Y$ is convex and bounded with diameter $D$.

For $T$ iterations:

• Maintain $r_k = \max\{ r_{k-1}, \|x_k - x_0\| \}$,
• $p_k = p_{k-1} + r_k^2 \|s_f(x_k)\|^2$; $s_f(x_k) \in \partial f(x_k)$,
• $\alpha_k = r_k^2 / \sqrt{p_k}$,
• Compute $v_{k+1} = \Pi_Y[x_k - \alpha_k s_f(x_k)]$,
• Randomized feasibility update as above.

A weighted average output $\bar{x}_\tau$ minimizes $r_{k+1}^2/\sum_{i=1}^k r_i^2$.

4. Convergence Guarantees and Theoretical Rates

Strongly Convex, Smooth Case

For adaptive stepsizes as above and exponential weighting,

$\mathbb{E}[|f(\bar{x}_k) - f(x^*)|] \le \epsilon$

after $k = O(\log(1/\epsilon))$ outer iterations, provided the mean reduction in infeasibility per iteration meets a prescribed threshold (Chakraborty et al., 27 Jan 2026).

Convex, Possibly Nonsmooth Case

After $T$ iterations using DoWS with feasibility, the output $\bar{x}_\tau$ satisfies

$$\mathbb{E}[|f(\bar{x}_\tau) - f^*|] \le \max\{A_1(T), \min\{A_2(\tau),A_3(T)\}\},$$

with \begin{align*} A_1(T) &= \frac{2 D M_f}{\sqrt{T}}\left(\frac{D}{r}\right)^{\frac{2}{T}\ln(e D^2/r^2)},\ A_2(\tau) &= D M_f \max_{1\le k\le\tau} \mathbb{E}[ (1-q)^{N_k/2} ],\ A_3(T) &= \frac{D M_f}{T} \left(\frac{D}{r}\right)^{\frac{2}{T}\ln(e D^2/r^2)} \sum_{k=1}^{\tau} \mathbb{E}[(1-q)^{N_k/2}], \end{align*} yielding the optimal $O(1/\sqrt{T})$ rate as $T \to \infty$ up to sampling-determined terms.

For unbounded $Y$, a tamed (logarithmically adjusted) variant of the DoWS step-size ensures bounded iterates and the same $O(1/\sqrt{T})$ expected error rate up to constants that grow logarithmically in $T$.

5. Sampling Distribution Regimes and Computational Properties

Performance and theoretical rates depend critically on the sampling distribution of the number of feasibility substeps $N_k$ at each outer iteration. For common regimes:

• Deterministic polynomial growth: $N_k = \lceil k^{1/p} \rceil$ ensures that the sum $\sum (1-q)^{k^{1/(2p)}}$ is uniformly bounded.
• Poisson sampling: $N_k \sim \mathrm{Pois}(\lambda_k)$ with $\lambda_k \approx k^{1/p}$ yields $$\mathbb{E}[(1-q)^{N_k/2}] = \exp(-\lambda_k(1-\sqrt{1-q}))$$, which decays polynomially in $k$.
• Binomial sampling: $N_k \sim \mathrm{Bin}(n_k, p)$ with $n_k \approx k^{1/p}$ gives similar decay properties.

Sub-polynomial growth of $N_k$ suffices to make sampling-driven error negligible at polylogarithmic cost in total feasibility steps.

6. Empirical Evaluation: QCQP and SVM Applications

Simulations were conducted on two canonical classes of problems:

Quadratically Constrained Quadratic Programming (QCQP)

The problem: $$\min_{x\in [-10,10]^{10}} x^\top A x + b^\top x \quad \text{s.t. } x^\top C_i x + u_i^\top x - e_i^\top x \le 0,\,\, i=1..m$$ was tested in three regimes:

• (a) Strongly convex $A \succ 0$, known $f^*$,
• (b) Strongly convex, unknown $f^*$,
• (c) Convex $A \succeq 0$, unknown $f^*$.

Baselines included the Nedić et al subgradient-projection, Arrow–Hurwicz and Alt-GDA primal-dual schemes, ACVI (ADMM+log-barrier), and CVXPY interior-point.

Key observations:

• Adaptive Polyak-step algorithm achieved linear convergence in (a), requiring no prior knowledge of strong convexity or smoothness parameters.
• DoWS and T-DoWS performed competitively in (b), (c), attaining the expected $O(1/\sqrt{T})$ rate slope.
• ACVI provided the fastest infeasibility decay but required expensive tuning.

Support Vector Machine (SVM) Soft-Margin Classification

For the SVM problem

$$\min_{w,b,\xi} \frac{1}{2}\|w\|^2 + C \sum_i \xi_i \quad \text{s.t. } 1 - \xi_i - y_i(w^\top z_i + b) \le 0,\,\, \xi_i \ge 0,$$

the UCI Banknote, Breast-Cancer, and MNIST 3-vs-5 datasets were used. Only DoWS/T-DoWS and primal-dual (Arrow–Hurwicz/Alt-GDA) baselines were compared due to convexity.

Results:

• DoWS/T-DoWS schemes reduced objective and infeasibility rapidly;
• Test-set misclassification rates were competitive with cross-validated primal-dual methods;
• Methods required no parameter tuning.

7. Theoretical Significance and Practical Implications

Randomized feasibility algorithms with Polyak steps provide a rigorously justified, computation-efficient approach to large-scale constrained convex optimization where projection onto intersected constraints is intractable. Theoretical results guarantee:

• Linear convergence to any prespecified tolerance for strongly convex, $L$-smooth $f$;
• Optimal $O(1/\sqrt{T})$ rates in the convex, potentially nonsmooth setting;
• Bounded sampling-driven error without demanding hyperparameter tuning or explicit knowledge of problem parameters.

Empirical results indicate practical competitiveness against state-of-the-art first-order and primal-dual methods, particularly when problem structure or scale make conventional projection approaches prohibitively costly (Chakraborty et al., 27 Jan 2026).