Alternating Optimization (AO)

• Alternating Optimization (AO) is a decomposition technique that splits interdependent variables into blocks for iterative, tractable subproblem optimization.
• It leverages structured subproblems and advanced strategies like ADMM, SCA, and manifold projections to efficiently handle nonconvex constraints.
• Widely applied in wireless communications, tensor factorization, and image restoration, AO offers theoretical guarantees on convergence under specific conditions.

Alternating Optimization (AO) is a fundamental decomposition principle for solving complex optimization problems involving multiple interdependent variables, often under nonconvex and structured constraints. The essential idea is to partition the parameter space into blocks and iteratively optimize each block while keeping the others fixed, thereby transforming an intractable global problem into a sequence of more manageable subproblems. This strategy has broad impact across signal processing, tensor decomposition, wireless communications, adaptive filtering, nonconvex statistical estimation, and machine learning.

1. Mathematical Framework and Algorithmic Structure

Consider an objective function $f(z_1, \ldots, z_B)$, defined over a product space of variable blocks $z_b \in \mathcal Z_b$. The classical AO iteration at step $t$ proceeds by cycling through blocks: $$z_b^{(t+1)} = \arg\min_{z_b \in \mathcal Z_b} f\left(z_1^{(t+1)}, \ldots, z_{b-1}^{(t+1)}, z_b, z_{b+1}^{(t)}, \ldots, z_B^{(t)}\right), \quad b=1,\ldots,B.$$ Each subproblem typically leverages the structure of $f$ and may admit closed-form or efficiently solvable updates, particularly when the block-wise subobjective is convex. In high-dimensional settings or problems with intricate constraints (e.g., unit-modulus, sparsity, manifold structure), AO provides tractable decomposition while retaining global monotonic descent properties (Ha et al., 2017, Ono et al., 2017, Murdoch et al., 2014).

Several advanced AO schemes couple the above with additional optimization primitives:

• AO-ADMM: Embeds proximal/augmented Lagrangian ADMM substeps in each block update to accommodate hard and soft constraints, e.g., in tensor models or matrix reconstruction (Li et al., 2014, Roald et al., 2021).
• AO-SCA: Use of Successive Convex Approximation to handle highly nonconvex block subproblems (Zhou et al., 28 Apr 2025).
• AO with Structured Manifolds: Projection and search on Riemannian or complex circle manifolds for constraints such as unit modulus (Lee et al., 2024, Bahingayi et al., 21 Aug 2025).
• AO with Subspace Escape: Employs higher-dimensional subspace searches to escape saddle points and poor local minima, as in Expanded AO (Murdoch et al., 2014).

The convergence trajectory is typically characterized by monotonic improvement in the objective and convergence to stationary/KKT points under broad conditions.

2. Convergence Properties and Theoretical Insights

Convergence of AO—especially in the nonconvex or constrained regime—has been rigorously analyzed using the framework of local concavity coefficients and restricted strong convexity/smoothness conditions (Ha et al., 2017). Key conclusions include:

• For exact minimization of each block, AO attains linear convergence in objective value within a local neighborhood, contracting at a rate

$$\rho = \left(1 - \frac{\alpha_x}{2\beta_x}\right)^{1/2} \left(1 - \frac{\alpha_y}{2\beta_y}\right)^{1/2},$$

where $(\alpha_b, \beta_b)$ are block-wise RSC/RSM constants. This rate depends on the better conditioned block, in contrast to joint gradient descent, which depends on the worst block (Ha et al., 2017).

• Monotonic descent to a stationary point is guaranteed if each subproblem is solved exactly or with diminishing errors. Inexact AO (e.g., alternating projected-gradient or iterative thresholding) remains convergent to a neighborhood of the optimum as long as error control matches contraction induced by local strong convexity (Ha et al., 2017, Ono et al., 2017).
• Expanded AO schemes that explore additional subspaces guarantee improved escape from suboptimal stationary points, though global guarantees rely on the subspace dimension and geometry of the nonconvex landscape (Murdoch et al., 2014).
• In the presence of nonconvex or non-Euclidean feasible sets (e.g., rank or sparsity constraints), convergence is controlled by local curvature and conditioning, as quantified by local concavity coefficients $\gamma_z(\mathcal Z)$ (Ha et al., 2017).

3. Applications and Model Classes

Alternating optimization underpins a diverse portfolio of applications, with custom algorithmic schemes tailored for each domain:

• Wireless Communications: AO enables tractable joint optimization in MIMO/MISO systems employing intelligent reconfigurable surfaces (IRS/SIM), where beamforming and phase-shift parameters are coupled and constrained. Here, AO cycles between digital beamforming optimization (e.g., via WMMSE or QCQP) and manifold-constrained phase shift updates, with subproblems further benefited by SCA or low-complexity closed-form solutions (Zhou et al., 28 Apr 2025, Bahingayi et al., 21 Aug 2025, Lee et al., 2024).
• Tensor and Matrix Factorization: AO, especially with ALS or embedded primal-dual/adaptive splitting, is crucial for fitting CP, PARAFAC2, and regularized decompositions. AO strategies manage nonnegativity, sparsity, TV, and other regularizations by alternating between factor updates and dual/penalty variable updates (via ADMM or PDS) (Ono et al., 2017, Roald et al., 2021).
• Sparse Adaptive Filtering: AO with shrinkage-based alternating updates rapidly identifies the support of time-varying sparse filters, outperforming standard single-stage shrinkage in both convergence and adaptation, as demonstrated theoretically (mean-square error recursions) and via simulation (Lamare et al., 2014).
• Image Restoration: Poissonian and other structured noise models benefit from AO schemes that split the optimization over data-fidelity (e.g., Poisson likelihood) and regularizer terms (TV, wavelets), each handled by convex proximal solvers or TV fixed-point iterations (Figueiredo et al., 2010).
• Nonconvex Statistical Estimation: In matrix completion, RPCA, penalized regression (e.g., with MCP, $\ell_0$), and multitask regression, AO exploits block separability and produces fast contraction in ill-conditioned or saddle-ridden regimes where joint descent is hampered by the worst-conditioned block (Ha et al., 2017, Murdoch et al., 2014).

4. Algorithmic Refinements and Practical Implementation

Substantial performance improvement in AO arises from advanced implementation strategies:

• Ordering of Block Updates: In IRS/SIM-aided downlink sum rate maximization, reordering AO to update phase shifts before digital beamforming markedly increases achievable rates (up to 115.53% in practical scenarios), due to more effective channel shaping (Bahingayi et al., 21 Aug 2025).
• Inner Iteration Depth: Running multiple projected-gradients for nonconvex, manifold-constrained subproblems (rather than single-shot updates) consistently attains higher-quality optima and avoids fast saturation behavior (Bahingayi et al., 21 Aug 2025).
• Surrogate and Relaxed Updates: SCA and first-order Taylor expansions are used to design efficiently solvable block subproblems and to derive closed-form per-element updates under unit-modulus or similar constraints, drastically reducing complexity while preserving accuracy (Zhou et al., 28 Apr 2025, Lee et al., 2024).
• Embedding ADMM/Primal-Dual Splitting: For structured regularizations where direct block minimization is prohibitive, nested ADMM or primal-dual splitting inside AO enables broad constraint handling and inversion-free updates, leading to faster and more flexible algorithms compared to inversion-based AO-ADMM (Ono et al., 2017, Li et al., 2014, Roald et al., 2021).
• Adaptive/Oracle-guided Blocks: In adaptive filtering, AO alternates between coefficient and support estimation, yielding near-oracle MSE performance and rapid adaptation to non-stationary sparsity patterns (Lamare et al., 2014).

5. Empirical Performance and Complexity

AO frameworks have demonstrated robust empirical performance, with convergence attained in a modest number of outer iterations (often $\leq$10–15 in modern wireless or tensor factorization applications (Zhou et al., 28 Apr 2025, Lee et al., 2024)). AO-embedded primal-dual approaches run up to $3\times$ faster and achieve lower error than matrix-inversion-based methods in constrained CPD (Ono et al., 2017). In structured matrix recovery, hybrid AO-ADMM techniques approach oracle performance and SNR-limited Cramér–Rao bounds (Li et al., 2014).

Complexity per iteration is primarily governed by the cost of block updates; for instance, solving convex QCQP or semidefinite relaxations for phase optimization may scale as $O((KN)^3)$ per iteration in massive MIMO (Zhou et al., 28 Apr 2025). Low-complexity surrogates and per-element methods offer significant reductions.

A sample complexity-complexity summary based on available data is below:

Domain	AO Step Complexity	# Iterations to Convergence	Reference
XL-IRS design	$O((KM)^3)$ (beamforming)	$\leq$15	(Zhou et al., 28 Apr 2025)
Tensor CPD (AO-PDS)	$O(N_d R)$ per block per inner step	$3\times$ speedup over AO-ADMM	(Ono et al., 2017)
Manifold phase AO	$O(N^3L)$ per outer iteration	$\leq$10	(Lee et al., 2024)

6. Expanded AO and Escape Strategies

Standard AO may be trapped at inferior local minima or stationary points due to the inherently limited 1D or blockwise nature of the search. Expanded AO schemes introduce “escaping” subspace searches over multidimensional affine spans (e.g., scaling, joint block updates, or greedy directions), interleaved with standard AO sweeps (Murdoch et al., 2014). Empirical evidence from large-scale matrix factorization and nonconvex regression demonstrates up to $10\times$ faster convergence and significant reductions in error by including judiciously designed escape phases. However, these improvements require careful subspace selection and moderate additional overhead per AO cycle.

A plausible implication is that, in highly nonconvex landscapes or when stationary points are known to be prevalent, algorithms that combine standard AO with periodic subspace escapes can be expected to deliver strictly improved empirical performance, provided subspace dimension is chosen to balance computational cost with escape efficacy (Murdoch et al., 2014).

7. Limitations and Domains of Effectiveness

While AO is broadly applicable, certain classes of problems remain challenging:

• Absence of blockwise tractability: If block updates cannot be solved efficiently (e.g., lack of closed-form, nonconvex constraints without cheap prox), AO may not offer complexity benefits.
• Pathological nonconvex landscapes: Like most local optimization methods, AO gives no global optimality guarantee; convergence is to stationary points or KKT points.
• Ill-conditioning in all blocks: If all blocks are severely ill-conditioned, the overall rate of AO may be only marginally better than joint methods (Ha et al., 2017).
• Proper initialization: Local convergence rates assume initialization in a suitable neighborhood; for arbitrary or random starts, global behavior cannot be ensured without additional randomization or escape strategies.

Nevertheless, the persistent empirical and theoretical gains derived from the blockwise minimization approach and the ease of integration with convex-relaxation, variable splitting, and gradient-type subroutines affirm AO as a densely interconnected methodological cornerstone across optimization-centric research disciplines.