AO-ADMM: Alternating Optimization with ADMM

• AO-ADMM is a versatile framework that decomposes complex optimization problems into block updates solved via ADMM, efficiently handling non-smooth penalties and hard constraints.
• It leverages adaptive penalty updates, warm starting, and FFT-based computation to accelerate convergence in high-dimensional and real-time applications.
• The framework supports diverse applications including matrix/tensor factorization, PARAFAC2 decomposition, and online adaptive control in multi-agent and imaging systems.

The term AO-ADMM refers to a family of algorithmic frameworks that combine alternated optimization (AO) or alternating minimization with the alternating direction method of multipliers (ADMM) as the workhorse subsolver for block updates. AO-ADMM architectures are broadly used in modern computational optimization, signal processing, machine learning, and multi-modal tensor decompositions, where non-smooth penalties, hard constraints, and non-convex structures are simultaneously present. The essential idea is to decompose a complex optimization problem into (possibly several) blocks, optimize each block in turn (the AO structure), and solve each block subproblem to high (or controlled) precision using ADMM, fully leveraging ADMM’s flexibility for splitting non-smooth constraints and general penalties. The term encompasses both classical block-separable designs in matrix/tensor factorization as well as several forms of adaptive, extrapolated, or online ADMM algorithms whose innovation lies in adaptive penalty updates, blockwise primal/dual variable splitting, or composite convexity.

1. General AO-ADMM Framework for Constrained Factorizations

The AO-ADMM framework in matrix and tensor factorization seeks to minimize an objective of the form

$$\min_{H_1, \ldots, H_N}\; l\big(Y - [H_d]_{d=1}^N\big) + \sum_{d=1}^N r_d(H_d)$$

where $l(\cdot)$ is a general (typically separable) loss and the $r_d$ are proximable regularizers or indicator functions for block constraints (Huang et al., 2015).

The algorithm alternates over blocks $H_d$, solving each with the other blocks held fixed. For each block, one forms a subproblem

$\min_{H}~ l(Y - W H^\top) + r(H)$

where $W$ represents the current values of the other factors. This subproblem is then solved via ADMM by introducing auxiliary variables to decouple loss and constraints/regularization. When the loss is least-squares, the resulting ADMM steps involve: (1) solving a regularized normal equation for the block, (2) a proximity update for the constraint/regularizer, and (3) a dual variable update.

A key feature of this framework is that almost any loss function and blockwise convex constraint (including nonnegativity, simplex constraints, or sparsity) can be accommodated so long as the proximal operator is efficiently computable.

Summary of Algorithmic Structure

Step	Operation	Domain of Application
Outer AO Loop	Over blocks/factors	All blockwise-separable problems
ADMM Subproblem	Block update via variable splitting	Each block update, decoupling constraints
Proximal Mapping	General penalty/constraint via prox operator	Nonnegativity, $\ell_1$-penalty, etc.

Computation caching, warm starting, and tuning of penalty and relaxation parameters are essential for fast convergence, especially in high-dimensional applications. The outer AO framework ensures that every limit point is a stationary point in the block-coordinate sense when standard BCD theory applies. Practical routines for non-negative matrix/tensor factorization, constrained completion, and dictionary learning are all captured within this general framework (Huang et al., 2015).

2. AO-ADMM for Tensor Models and PARAFAC2

AO-ADMM, as applied to the PARAFAC2 tensor model, enables direct handling of richly constrained factorizations in which the factor corresponding to one mode is allowed to evolve across slices, subject to a "constant cross-product" constraint. Existing ALS-based (alternating least squares) methods cannot directly impose general regularization on the evolving factors. AO-ADMM overcomes this by introducing explicit auxiliary variables for both the regularization penalties and the coupling constraint, formulating an augmented Lagrangian and using ADMM for block updates (Roald et al., 2021, Roald et al., 2021).

The algorithm cycles through blockwise updates for each mode (A-mode, B-mode, D-mode), applying an ADMM inner loop to decouple data fidelity, proximable penalties (e.g., nonnegativity, graph Laplacian, total variation), and the PARAFAC2 constraint. For the evolving factors, the key step is a projection operation involving an alternating Procrustes/SVD scheme to enforce the cross-product constraint, while proximal operators handle additional penalties. Empirically, AO-ADMM enables the inclusion of arbitrary proximable penalties or hard constraints on all modes, improves interpretability in practical data mining and chemometrics tasks, and converges faster or with higher final accuracy than unconstrained ALS or flexible coupling HALS methods (Roald et al., 2021).

Key Algorithmic Elements in PARAFAC2 AO-ADMM

Component	Description
AO Outer Loop	Alternates between A, B, D modes
ADMM Substeps	Primal (block) update, constraint projection, dual update
Evolving Factors	Projection onto constant cross-product via SVD-based coupling
Penalties	Any proximable function (nonnegativity, TV, smoothing, etc.)

3. Adaptive and Online AO-ADMM Algorithms

Recent advances have focused on adaptivity in AO-ADMM, including both adaptive selection of penalty parameters and online processing suitable for dynamic or decentralized systems. These include:

• "An Adaptive Alternating Direction Method of Multipliers" (aADMM), which generalizes the classical ADMM to settings in which $f$ is $\alpha$-convex and $g$ is $l(\cdot)$0-convex (possibly weakly convex if $l(\cdot)$1). The algorithm employs adaptive penalties $l(\cdot)$2 chosen based on the generalized convexity parameters to ensure convergence, relying on a duality with the adaptive Douglas–Rachford algorithm. The adaptive scheme is essential to maintain convergence in the presence of non-strongly convex and/or weakly convex components (Bartz et al., 2021).
• "Flexible MPC-based Conflict Resolution Using Online Adaptive ADMM" (OA-ADMM), in which a penalty vector $l(\cdot)$3 is updated online according to problem dynamics, with a similarity ("forgetting") factor $l(\cdot)$4 incorporated in the dual update for adaptation to time-varying environments. This structure is particularly suited to distributed real-time control and model predictive control in autonomous vehicle coordination, allowing each agent to adapt penalties in response to evolving collision-avoidance requirements (An et al., 2021).

These adaptive AO-ADMM architectures are designed to meet stringent robustness or real-time performance requirements absent in the classical fixed-penalty setting.

4. AO-ADMM with Trajectory-Following Acceleration

The AO-ADMM concept also encompasses a class of adaptive-acceleration schemes. In "Trajectory of Alternating Direction Method of Multipliers and Adaptive Acceleration" (Poon et al., 2019), AO-ADMM denotes adaptive trajectory-following extrapolation within ADMM, distinct from classical inertial acceleration. When the ADMM dual fixed point evolves along a spiral, standard inertial methods are ineffective or detrimental. AO-ADMM instead collects a sequence of difference vectors, fits a linear predictive model, and extrapolates $l(\cdot)$5 steps along the locally linearized trajectory, damping the step for global convergence. The approach generalizes minimal polynomial or reduced-rank extrapolation, and is triggered only when the local model is contractive. Theoretical analysis establishes that, under a summable sequence of local perturbations, global convergence to a saddle point is preserved, and local convergence can be accelerated by reducing the effective spectral radius from $l(\cdot)$6 to $l(\cdot)$7 of the local update operator.

Key experimental results suggest significant reductions (factor $l(\cdot)$8–$l(\cdot)$9) in iteration count versus standard ADMM across problems including $r_d$0-minimization and TV inpainting, with negligible computational overhead for modest extrapolation order $r_d$1 (Poon et al., 2019).

5. AO-ADMM in Imaging and Hybrid Optimization

AO-ADMM provides a flexible platform for composite inverse problems in imaging, such as myopic deconvolution under total variation (TV) regularization (Chen et al., 2020). In these settings, the algorithm splits variables (e.g., image $r_d$2 and point spread function $r_d$3), applies an outer ADMM to decouple non-smooth TV terms via auxiliary variables, and employs a specialized inner solver (e.g., Linearize And Project, or LAP) for data-fidelity subproblems that are tightly coupled and bound-constrained. The practical efficiency of AO-ADMM arises from the FFT-accelerated structure of image operators and efficient blockwise shrinkage on auxiliary variables. Convergence to stationary points is established under standard conditions, and the AO-ADMM (ADMM-LAP) method outperforms block coordinate descent ADMM both in convergence speed and final reconstruction accuracy in adaptive optics retinal imaging benchmarks (Chen et al., 2020).

6. Computational Complexity and Implementation Considerations

The computational efficiency of AO-ADMM architectures hinges on the following principles (Huang et al., 2015, Roald et al., 2021):

• Each ADMM subproblem can exploit precomputed matrix factorizations (e.g., Cholesky of $r_d$4 in matrix factorization) and warm starting, reducing the cost per iteration.
• For tensor models, per-AO-iteration cost is dominated by ridge-regularized linear solves for each block—e.g., $r_d$5 for PARAFAC2.
• For imaging, each outer ADMM iteration often costs $r_d$6 due to FFT-based acceleration for images of size $r_d$7.
• Adaptive penalty selection and careful tuning are critical to maintain stability, convergence rate, and numerical robustness, especially in the presence of ill-conditioning, non-strong convexity, or time-varying/online constraints.
• Empirically, AO-ADMM methods typically converge in fewer outer iterations than pure ALS/BCD or block coordinate descent ADMM methods for the same class of problems, even when only a small number of inner ADMM substeps are performed per block.

7. Applications and Empirical Results

AO-ADMM algorithms have demonstrated strong empirical performance across a range of domains:

• In nonnegative and constrained tensor decomposition (PARAFAC2), AO-ADMM achieves higher factor match scores (e.g., median FMS 0.96 for unimodality-constrained $r_d$8 factors compared to 0.82 for unconstrained) and fast convergence—typically a handful of outer iterations—while supporting arbitrary proximable penalties and hard constraints (Roald et al., 2021).
• In real-world neuroscience (multi-subject fMRI) and chemometric (GC-MS) datasets, AO-ADMM constraints yield more interpretable and physically realistic solutions (e.g., nonnegative, smooth, or piecewise-constant factors).
• In decentralized multi-agent control (autonomous vehicles), online AO-ADMM achieves a 47.93% reduction in mean added delay compared to competing methods, while ensuring real-time safe operation via adaptive penalties (An et al., 2021).
• In myopic deconvolution for adaptive optics retinal imaging, ADMM-LAP (an AO-ADMM instance) converges in fewer iterations, achieves higher SNR restorations, and runs $r_d$9–$H_d$0 faster than a comparable block coordinate ADMM solver (Chen et al., 2020).

These empirical findings underscore the broad utility and practical advantages of AO-ADMM frameworks in handling composite, constrained, and non-smooth optimization problems in both batch and online contexts.