Alternating Iterative Projection Algorithms

• Alternating iterative projection algorithms are iterative methods that project onto intersecting sets to solve feasibility, optimization, and inverse problems.
• They employ cyclic, randomized, or block projection strategies, leveraging geometrical and spectral properties to ensure convergence under various conditions.
• These methods are widely applied in signal processing, compressive sensing, distributed optimization, and inverse problems, offering robust and efficient solutions.

An Alternating Iterative Projection Algorithm refers to a class of methods for solving feasibility, optimization, or inverse problems by iteratively projecting onto multiple sets—usually subspaces, affine spaces, or more general convex or nonconvex sets—by alternating application of projection (or proximal) operators, possibly under randomized, block, or distributed control. These methods have deep roots in classical analysis and geometry, with modern extensions underpinning a wide range of applications in signal processing, machine learning, distributed optimization, and inverse problems. The core idea is to obtain a solution in the intersection (possibly empty) of two or more sets by repeated projection, with the sequence of iterates governed by the geometry of the sets, the algebraic structure of the projections, and the choice of update order, either deterministic or randomized.

1. Foundations of Alternating Iterative Projection Algorithms

The canonical setting for alternating iterative projections is the convex feasibility problem: given closed convex sets $A, B \subset \mathcal{H}$ (Hilbert space), find $x^* \in A \cap B$ if possible. The classical von Neumann algorithm alternates projections as

$$\begin{cases} x_{n+1} = P_A(y_n), \ y_{n+1} = P_B(x_{n+1}), \end{cases}$$

where $P_A, P_B$ are the respective nearest-point projections. For two closed subspaces, this sequence converges in norm to the projection of the initial point onto $A \cap B$, as shown by the original von Neumann theorem and refined by subsequent work on periodic and quasiperiodic ordering for families of subspaces (Ginat, 2018).

Generalizations have incorporated nonconvex sets, more than two sets, or various relaxation and acceleration schemes:

• Cyclic or non-cyclic projections for multiple closed convex sets.
• Dykstra's algorithm, extending alternating projections to intersections of many convex sets via auxiliary variables (Wei et al., 2015).
• Proximal, Bregman, or reflected projections (e.g., Douglas–Rachford splits) for monotone operator or nonconvex feasibility problems.

For optimization problems, alternating projections frequently manifest as block coordinate descent (BCD), sketch-and-project, or operator splitting algorithms. Notably, the structure can accommodate data-fitting terms (e.g., linear equations), structural constraints (e.g., sparsity, nonnegativity), and regularization (e.g., TV, $\ell_1$).

2. Algorithmic Structure and General Methodology

The alternating iterative projection paradigm has several prominent instantiations:

a. Binary Case (Two Sets)

Given $A$, $B$, iteratively compute

$x_{n+1} = P_A(P_B(x_n)).$

For convex $A$, $B$, this sequence has strong convergence guarantees. In the nonconvex case, convergence is subtler, depending on geometric and analytic properties such as the three-point property, local contraction, and the Kurdyka–Łojasiewicz (KL) property (Zhu et al., 2018).

b. Block and Randomized Extensions

For $m$ sets $A_1, \ldots, A_m$, use cyclic or random ordering: $x_{n+1} = P_{A_{i_n}}(x_n)$ with $(i_n)$ a periodic or randomized sequence. Block-structured methods project onto intersections or combine projections using auxiliary correction terms (e.g., Dykstra’s algorithm) (Xiang et al., 2017), with convergence rates depending on spectral properties and operator geometry.

c. Algebraic and Operator Formulations

Many modern randomized iterative solvers for linear systems, such as Kaczmarz, coordinate descent, and their block/Gaussian variants, are unified as alternating iterative projection algorithms. They adopt the template: $x^{k+1} = x^k + \Xi_k(b - A x^k)$ where $\Xi_k$ is a (possibly randomized) projector depending on "sketch" matrices, and convergence is governed by spectral contraction properties (Xiang et al., 2017).

3. Convergence Properties and Theoretical Guarantees

Convergence analysis hinges on the properties of the sets and projections, the underlying space, and the update mechanism:

• Convex Case: For two closed convex sets in Hilbert space, norm convergence (strong) is assured under periodic orderings (Halperin's theorem), with geometric convergence rate determined by the cosine of the minimal angle between subspaces. For nonperiodic or arbitrary orderings, weak convergence still holds (Amemiya–Ando theorem), but norm convergence may fail in infinite dimensions for pathological sequences (Ginat, 2018).
• Nonconvex/Structured Case: If the sets satisfy quantitative geometric properties, such as the three-point and local contraction properties and have semi-algebraic or definable structure, then the alternating projection scheme converges to a critical point, with linear or sublinear rate determined by the KL exponent at the solution (Zhu et al., 2018). This framework encompasses many structured inverse problems.
• Banach Spaces: Under suitable uniform convexity and smoothness—as quantified by power-type moduli or the Bregman/proximal setup—alternating projections admit linear convergence, provided additional regularity on the sum or intersection of subspaces (Bargetz et al., 2019).
• Randomized and Block Methods: Linear convergence rates for randomized block-projection schemes are governed not by the squared Frobenius norm of $A$ but by extremal singular values or global condition numbers, leading to dimension- and condition-number–controlled rates (Xiang et al., 2017).

4. Applications Across Domains

Alternating iterative projection algorithms have been broadly deployed:

• Signal and Image Processing, Inverse Problems: Image reconstruction (ptychography, phase retrieval) uses AP for enforcing constraints in data and transform domains (Marchesini et al., 2014), with convergence driven by problem geometry and often enhanced by spectral-phase synchronization initializations.
• Compressive Sensing: Generalized AP (GAP) and its extensions alternate between data-fidelity projections and structural/sparsity constraints (e.g., TV minimization), achieving linear or near-linear convergence rates under restricted isometry conditions (Yuan, 2015, Yuan et al., 2015).
• Distributed Optimization/Consensus: Distributed min–max convex optimization and time-optimal multi-agent consensus are formulated via epigraph projections, implemented through outer-AP (Bregman) and inner intersections (Dykstra’s method) in networked settings (Hu et al., 2014).
• Machine Learning and Gaussian Processes: Large-scale GP training and inference with ill-conditioned kernel matrices are handled via block AP, achieving $\mathcal{O}(n)$ memory scaling and linear convergence, in contrast to quadratic-scaling CG solvers (Wu et al., 2023).
• Array Signal Processing/Neuroimaging: Alternating projections are employed for MEG/EEG source localization, directly minimizing LS/ML cost by sequentially reoptimizing dipole locations and waveforms, exhibiting superior robustness to source correlation and forward-model errors (Adler et al., 2019, Adler et al., 2022).
• Tomography and Linear Algebra: Kaczmarz’s and randomized block Kaczmarz methods iteratively project onto the solution sets of single or block rows of linear systems, with convergence rates dictated by subspace geometry and randomized operator expectations (Wallace et al., 2014, Xiang et al., 2017).

A table summarizing core AP algorithmic variants:

Algorithmic setting	Projection type / Order	Convergence Regime
Two convex sets	Alternating nearest-point	Norm (strong), linear or sublinear
Block/cyclic/randomized	Cyclic, randomized, block	Linear, rate by geometry/condition
Nonconvex/semi-algebraic	Alternating with KL/3-pt	To critical point, KL-dependent
Distributed/epigraph	Outer-AP + Dykstra inner	Linear, per the projections

5. Accelerations, Extensions, and Robustness

Several modern trends and theoretical developments are prominent:

• Randomization/Block-Selection: Selecting sketch/coordinate blocks or rows at random or by Gauss–Southwell criteria can reduce iteration counts and improve scalability, especially for large or coherent systems (Wu et al., 2023, Xiang et al., 2017).
• Dykstra and Hybrid Projections: Dykstra's alternating projection enables rigorous handling of intersections of many convex sets, as in fully constrained least-squares or spectral unmixing (Wei et al., 2015). Convex–combination schemes blend different fixed-point operators, such as Douglas–Rachford and AP, improving empirical robustness and convergence on nonconvex problems (Thao et al., 2020).
• Robustness to Perturbations: Stability of convergence under set-approximation and modeling error is established using the Attouch–Wets convergence framework, with geometric criteria such as strong exposure, full interior, or subspace closure guaranteeing robustness (Bernardi et al., 2019).
• Spectral and Phase-Synchronization Enhancements: In high-dimensional phase retrieval and imaging, spectral GCL-based initialization provides near-optimal phase estimates, accelerating subsequent AP convergence (Marchesini et al., 2014).

6. Quantitative Rates, Limitations, and Comparative Performance

Sharp quantitative convergence results are known for structured settings:

• Linear Convergence: For convex (Hilbert space/subspace) settings or under strong regularity/KL property, strictly linear rates can be proved, with the contraction factor determined by principal subspace angles, restricted isometry constants, or spectrum of block subproblems (Bargetz et al., 2019, Ginat, 2018).
• Sublinear and Slow Regimes: In tangential intersection cases or with flat epigraphs, AP can be arbitrarily slow—only logarithmic, or even slower—necessitating alternative algorithms (e.g., Douglas–Rachford, which often accelerates convergence in such cases) (Bauschke et al., 2015).
• Empirical Performance: Across domains, AP-based algorithms routinely outperform scanning, beamformer, or coordinate-descent approaches, especially under source correlation, ill-conditioning, or massive data regimes (Wu et al., 2023, Adler et al., 2022, Wei et al., 2015).

7. References to Leading Research and Further Reading

The theory and application of alternating iterative projection algorithms are substantially advanced in several key works:

• O. Ginat, "The Method of Alternating Projections," for theoretical and convergence properties (Ginat, 2018).
• Badea and Seifert, "Quantified asymptotic behaviour of Banach space operators..." for extension to Banach structures and asymptotic rates (Badea et al., 2017).
• Tan and Vershynin's framework for unifying randomized iterative projection methods (Xiang et al., 2017).
• Various algorithmic applications: Gaussian processes (Wu et al., 2023), compressive sensing (Yuan et al., 2015, Yuan, 2015), spectral unmixing (Wei et al., 2015), and neuroimaging (Adler et al., 2022).
• For convergence under perturbations: De Bernardi and Miglierina (Bernardi et al., 2019).
• Dykstra-based and hybrid algorithms: (Wei et al., 2015, Thao et al., 2020).

Alternating iterative projection algorithms thus comprise a versatile and theoretically rigorous framework, central to both foundational mathematical analysis and cutting-edge computational applications.