Generalized Approximate Message Passing (GAMP)

• GAMP is an iterative inference algorithm that decouples high-dimensional estimation into scalar problems using approximate message passing ideas.
• It extends AMP by handling arbitrary input and output distributions, enabling applications in compressed sensing, nonlinear recovery, and phase retrieval.
• Its performance is rigorously predicted through state evolution equations that track mean-squared error and detection thresholds in large random systems.

Generalized Approximate Message Passing (GAMP) is an iterative inference algorithm designed for high-dimensional estimation in systems where an unknown random vector is observed through a linear transform, potentially followed by a probabilistic and possibly nonlinear output channel. The algorithm extends the ideas of approximate message passing (AMP) to incorporate arbitrary (separable) input and output distributions and efficiently approximates both maximum a posteriori (MAP) and minimum mean-squared error (MMSE) inference. GAMP supports a wide range of problems, including compressed sensing with non-Gaussian priors, nonlinear measurement channels (e.g., quantization, phase retrieval), and high-dimensional regression with complex noise models. Its performance and algorithmic behavior are understood through a set of state evolution equations that rigorously characterize asymptotic mean-squared error and other empirical measures in the large system limit, when the measurement matrix is random i.i.d. Gaussian.

1. The General Estimation Model and Algorithmic Structure

The estimation problem addressed by GAMP is the following: an unknown signal $x = (x_1,\ldots,x_n)$ is generated with independent (or conditionally independent) prior distributions $p_{X|Q}(x_j|q_j)$, possibly parameterized by external variables $q_j$. The observation model is

$z = A x, \qquad y_i \sim p_{Y|Z}(y_i | z_i)$

where $A \in \mathbb{R}^{m \times n}$ is a known measurement matrix and the output channel $p_{Y|Z}$ can be arbitrary and possibly nonlinear (e.g., quantization, phase retrieval, binary classification).

GAMP provides scalable, iterative updates that “decouple” inference into tractable scalar estimation problems at each node, using summary statistics (means and variances) that propagate according to approximations justified by the central limit theorem in the large-system limit (Rangan, 2010).

There are two algorithmic flavors:

• Max-sum GAMP: Approximates MAP estimation.
• Sum-product GAMP: Approximates marginal inference for MMSE estimation.

In both versions, the per-iteration update sequence consists of:

1. “Linear” steps, propagating pseudo-data via $A$ and $A^T$,
2. “Nonlinear” steps, applying scalar estimation functions derived from the input prior and output channel,
3. Variance (or uncertainty) updates essential for proper correction (Onsager terms) and performance prediction.

2. Mathematical Formulation and Update Equations

The generalized update sequence for iteration $t$ for all $j = 1,\ldots, n$ and $i = 1,\ldots, m$ is given by:

1. Output linear step:

$$\hat{p}_i = (A \hat{x})_i - \tau^p_i s_i, \qquad \tau^p_i = \sum_j |A_{ij}|^2 \tau^x_j$$

2. Output nonlinear step:

$s_i = g_{\text{out}}(\hat{p}_i, y_i, \tau^p_i)$

where $g_{\text{out}}$ is a scalar function, MAP or MMSE estimator for $z_i$ given $y_i$ and the pseudo-observation.

3. Input linear step:

$$\hat{r}_j = \hat{x}_j + \tau^r_j (A^T s)_j, \qquad \frac{1}{\tau^r_j} = \sum_i |A_{ij}|^2 \tau^s_i$$

4. Input nonlinear step:

$$\hat{x}_j = g_{\text{in}}(\hat{r}_j, q_j, \tau^r_j)$$

$g_{\text{in}}$ is the scalar input MAP or MMSE estimator for $x_j$.

These updates generalize classical message passing and belief propagation to arbitrary separable input and output distributions.

Table: High-level Structure of GAMP Updates

Step	Scalar Function	Comment
Output nonlinear	$g_{\text{out}}$	MMSE or MAP for channel $p_{Y|Z}$
Input nonlinear	$g_{\text{in}}$	MMSE or MAP for prior $p_{X|Q}$
Onsager corrections	Uses estimated variances	Ensures correct asymptotic Gaussianity/decoupling

3. State Evolution and Theoretical Guarantees

The analysis of GAMP in the large-system limit ($n,m \to \infty$ with $m/n \to \beta$ fixed, $A$ i.i.d Gaussian) is given by state evolution (SE) equations (Rangan, 2010). At each iteration, the empirical distribution of the estimates matches that of a scalar equivalent model with Gaussian noise, and the mean-squared error and other metrics are tracked by a set of scalar recursions.

For sum-product GAMP,

$$\tau^{x, t+1} = \mathbb{E} \left[ \text{Var}(X | R^t) \right], \qquad R^t = X + \sqrt{\tau^{r, t}} W$$

where $W \sim \mathcal{N}(0,1)$, and analogous equations for output quantities. The SE equations predict performance (e.g., final MSE, detection accuracy) exactly, even for non-convex settings and arbitrary (non-Gaussian, quantized, or nonlinear) observation models.

The fixed points of the SE equations correspond to (and have been shown to match) replica predictions from statistical mechanics, providing rigorous justification matching earlier non-rigorous results.

4. Supported Problem Classes and Notable Algorithmic Adaptations

GAMP is structurally versatile and supports a broad span of models:

• Non-Gaussian priors: e.g., Bernoulli-Gaussian, Laplacian, mixture models, weighted spike-and-slab for model-based compressed sensing (Oxvig et al., 2018).
• Nonlinear output channels: e.g., quantization as in estimation from quantized samples (Kamilov et al., 2011), modulus nonlinearity for phase retrieval (Schniter et al., 2014), binary classification (logistic/probit) (Ziniel et al., 2014), clipping, or self-reset ADCs (Musa et al., 2018).
• Unknown model parameters: Extensions include built-in EM-based parameter learning (Li et al., 2015, Huang et al., 2016), and fully adaptive GAMP (Kamilov et al., 2012) that learn both prior and noise parameters during inference, with provable asymptotic consistency.

Algorithmic innovations such as adaptive damping and mean removal enhance GAMP robustness to non-ideal measurement matrices (e.g., non-zero mean, correlated, or ill-conditioned $A$) (Vila et al., 2014). For structured sparsity, GAMP supports non-i.i.d priors with entry-dependent weights (Oxvig et al., 2018), and for ill-conditioned or non-i.i.d. transformation matrices, generalized memory variants (e.g., VAMP, GMAMP) extend state evolution and Bayes-optimality (Tian et al., 2021, Schniter et al., 2016).

5. Performance Analysis and Empirical Results

Various works provide detailed evaluation of GAMP's performance:

• Sample complexity: In compressive sensing with sublinear sparsity ($k = o(n)$), Bayesian GAMP achieves sample complexity $M \gtrsim \delta^* k \log (n/k)$ with a sharp threshold $\delta^*$ determined by state evolution (Takeuchi, 2024).
• Noise robustness: GAMP remains effective for measurement channels with significant nonlinearity or noise, e.g., recovering $K$-sparse signals from modulus-only noisy Fourier measurements with measurement SNR of 30 dB, output SNR $\geq 28$ dB (Schniter et al., 2014).
• Algorithmic efficiency: For very large-scale problems ($n \gtrsim 10^4$), per-iteration complexity is $O(mn)$, and overall runtime outperforms convex and greedy methods by orders of magnitude, especially as $n$ increases (Schniter et al., 2014).
• Extensions to decentralized settings: In distributed tree-structured networks, decentralized GAMP with consensus propagation matches the fixed points and performance of centralized GAMP (Takeuchi, 2023).

Empirical benchmarking consistently demonstrates that GAMP’s empirical phase transitions and estimation error match the state evolution predictions closely under the prescribed conditions.

6. Practical Implementation Considerations

Algorithm selection and tuning:

• For i.i.d. Gaussian $A$, classical GAMP is Bayes-optimal.
• For structured $A$ (e.g., ill-conditioned, nonzero-mean), use adaptive damping/mean removal (Vila et al., 2014) or vector variants (VAMP) (Schniter et al., 2016).
• For unknown priors or noise statistics, integrate EM or fully Bayesian parameter updates (Li et al., 2015, Kamilov et al., 2012).
• For dependence on prior structure (e.g., spatial sparsity or non-uniform importance), implement weighted priors or model-based variants (Oxvig et al., 2018).

Convergence and stability: Divergence can occur under strong matrix correlations, rank-deficiency, or incorrect modeling; mitigations include damping, mean removal, and proper variance normalization (Vila et al., 2014, Tian et al., 2021).

Extensions and limitations: While GAMP is highly general, its theoretical guarantees and performance analyses rest on the i.i.d. Gaussian $A$ assumption; for more general matrices, memory-augmented variants or VAMP provide extensions at some additional computational cost (Tian et al., 2021).

7. Impact, Limitations, and Evolving Directions

GAMP links belief propagation, statistical physics, and convex optimization into a computationally efficient, theoretically well-understood framework. It serves as a backbone for high-dimensional inference tasks in compressed sensing, sparse learning, signal recovery, and machine learning, with rigorous state evolution providing both performance prediction and phase transition analysis. Extensions continue to address robustness (e.g., to model mismatch (Saglietti et al., 2019)), decentralized inference (Takeuchi, 2023), and applications with structured priors or bilinear models (Parker et al., 2015).

Open directions include further unification with survey propagation for glassy optimization landscapes (Saglietti et al., 2019), generalizing state evolution to broader matrix ensembles, and refined analyses of finite-sample/finite-iteration dynamics across measurement and channel models.