Connecting randomized iterative methods with Krylov subspaces

Abstract: Randomized iterative methods, such as the randomized Kaczmarz method, have gained significant attention for solving large-scale linear systems due to their simplicity and efficiency. Meanwhile, Krylov subspace methods have emerged as a powerful class of algorithms, known for their robust theoretical foundations and rapid convergence properties. Despite the individual successes of these two paradigms, their underlying connection has remained largely unexplored. In this paper, we develop a unified framework that bridges randomized iterative methods and Krylov subspace techniques, supported by both rigorous theoretical analysis and practical implementation. The core idea is to formulate each iteration as an adaptively weighted linear combination of the sketched normal vector and previous iterates, with the weights optimally determined via a projection-based mechanism. This formulation not only reveals how subspace techniques can enhance the efficiency of randomized iterative methods, but also enables the design of a new class of iterative-sketching-based Krylov subspace algorithms. We prove that our method converges linearly in expectation and validate our findings with numerical experiments.

• The paper presents a new affine subspace formulation that bridges low-memory randomized iterative methods and full-scale Krylov methods, demonstrating linear convergence.
• It introduces IS-Krylov algorithms using projection-based updates that balance memory usage with computational efficiency.
• Empirical evaluations show that increasing the memory parameter ℓ accelerates convergence and enhances robustness for large-scale, ill-conditioned systems.

Connecting Randomized Iterative Methods with Krylov Subspaces: An Expert Analysis

---

Introduction

This paper, "Connecting randomized iterative methods with Krylov subspaces" (2505.20602), investigates the foundational interplay between two classes of algorithms for large-scale linear systems: Randomized Iterative Methods (RIMs) and Krylov Subspace Methods (KSMs). The authors develop an affine subspace framework that naturally interpolates between these approaches by adaptively adjusting the history parameter $\ell$. Utilizing projection-based updates, they present theoretical analysis, efficient implementation schemes, and empirical justification, culminating in a class of iterative-sketching-based Krylov (IS-Krylov) algorithms.

---

Unified Affine Subspace Formulation

The authors define a unified iterative scheme for solving $Ax = b$ with $A \in \mathbb{R}^{m\times n}$:

$$x^{k+1} = \mathop{\arg\min}_{x \in \Pi_k} \|x - A^\dagger b\|_2^2,$$

where the affine subspace $\Pi_k$ is

$$\operatorname{aff}\left\{x^{j_k},\ldots,x^k, \ x^k - A^\top S_kS_k^\top (Ax^k - b)\right\},$$

with sketching matrices $S_k$ drawn from a probability space, and $j_k = \max\{k-\ell+1, 0\}$.

The truncation length $\ell$ controls the memory and nature of the method:

• $\ell=1$: Reduces to standard stochastic iterative schemes (e.g., Randomized Kaczmarz [strohmer2009randomized]), where only the most recent iterate and sketched gradient are used.
• $\ell=\infty$: Recovers a full Krylov subspace method, as the update utilizes the span of all past iterates and sketched gradients.

This interpolation builds a rigorous bridge between the randomized (low-memory, low-cost) updates and history-augmented, subspace-expanding Krylov schemes.

---

Algorithmic Realization and Complexity

Practical implementation is achieved via orthogonal projection calculations and leveraging structure for efficient computation. The core iteration reduces to solving a small, well-structured least-squares problem in the subspace generated by previous iterates and the current sketched normal.

Key elements:

• Memory-efficient storage and update of the relevant subspace, exploiting affine independence.
• Use of the Gram-Schmidt process and recursive matrix structures to economically compute the required orthogonal projections.
• Complexity per iteration is linear in $n$ and, for fixed $\ell$, avoids the quadratic or cubic costs typical in classical KSMs.

Algorithms are presented in two forms: a direct projection (RIM-AS) and an orthogonalized basis expansion (IS-Krylov).

---

Theoretical Properties

Linear Convergence

A central theorem establishes that the proposed methods converge linearly in expectation:

$$\mathbb{E} \|x^{k+1} - A^\dagger b\|_2^2 \leq (1 - \rho_\ell) \|x^{k} - A^\dagger b\|_2^2,$$

where $\rho_\ell$ is an explicit function of the properties of the sketching distribution and, crucially, grows (i.e., convergence accelerates) with $\ell$. For $\ell \geq \mathrm{rank}(A)$, the method attains exact solution in finite iterations without round-off.

Subspace Optimality and Acceleration

As $\ell$ increases, the method has access to richer subspace information—incorporating more orthogonal components of the error—leading to strict improvement in the worst-case contraction. This framework formally subsumes and generalizes stochastic momentum (e.g., ASHBM [zeng2024adaptive]), inertial, and subspace-augmented stochastic approaches.

Algorithmic Equivalence

The authors prove the equivalence between the direct affine method and the IS-Krylov scheme, establishing that the two implementations produce identical iterates when the same sequence of sketching matrices is used, but with potentially different practical cost structures.

---

Connection to Krylov Subspace Methods

When the probability space degenerates to deterministic identity sketching and $\ell \to \infty$, the method reconstructs the classical KSM structure:

$$x^0 + \mathcal{K}_{k+1}(A^\top A, \ A^\top(Ax^0 - b)),$$

where the iterates evolve in the Krylov subspace generated by the system normal equations.

More generally, with iterative sketching (e.g., random blocks, Gaussian, SRHT), the method yields a new class of "IS-Krylov" algorithms, where the subspace is stochastically refined at each iteration. This perspective produces stochastic analogs of classical algorithms (e.g., CGNE, GMRES) and naturally explains the effect of increasing memory (subspace dimension) on convergence.

---

Empirical Evaluation

Numerical experiments showcase:

• Robust acceleration from larger $\ell$ and appropriate block sizes $q$.
• IS-Krylov methods outperforming classical RIMs, momentum-based acceleration, and match or exceed standard direct solvers on sufficiently large problems.
• The partition-based IS-Krylov (IS-Krylov-PS) is competitive in both iterations and wall-clock time, especially for large, sparse, or ill-conditioned systems.

Block and sketch type selection impact the tradeoff between iteration-complexity and per-iteration cost, and the authors offer detailed quantitative guidance.

---

Practical and Theoretical Implications

This framework unifies and broadens the landscape of scalable iterative solvers:

• One can tune $\ell$ for a spectrum of regimes: low-memory/high-speed (small $\ell$) and rapid-convergence/high-accuracy (large $\ell$).
• Facilitates the design of hybrid stochastic-Krylov algorithms for high-dimensional or distributed systems, compatible with sketch-and-solve and other randomized NLA paradigms.
• The framework applies even when $A$ is rank-deficient, over-/under-determined, or where only block sampling/sketching is available.

Theoretically, it provides insight into momentum, subspace, and history-based acceleration in randomized settings.

---

Future Directions

Key areas for extension include:

• Theoretical characterization of the optimal $\ell$ as a function of system properties and sketching scheme.
• Extension to inconsistent or streaming systems (e.g., randomized coordinate descent as a dual analog).
• Connections with variable-probability/structured sketching, preconditioning, and application to broader operator equations or nonlinear setups.

---

Conclusion

The paper delivers a methodologically rigorous and practically effective synthesis of randomized iterative and Krylov-subspace methods via an affine subspace framework. The interpolation between these domains reveals new algorithmic classes with provable and empirically validated convergence benefits. The resulting IS-Krylov strategies advance the state-of-the-art for large-scale, ill-conditioned, and structured linear systems.

---