Learned Multi-Layer VAMP (LMLVAMP)

• LMLVAMP is a hybrid estimator that combines model-based inference with data-driven neural denoisers to recover signals from complex, nonlinear, and quantized measurements.
• It integrates spectral priors with neural network–based denoising in a message-passing framework, leveraging fast Fourier transforms for efficient processing.
• The approach achieves significant NMSE reductions and approaches oracle performance in FR3 6G scenarios, effectively mitigating interference, saturation, and quantization effects.

Learned Multi-Layer Vector Approximate Message Passing (LMLVAMP) is a hybrid model-based/data-driven estimator designed for recovering a desired signal from nonlinear, quantized receiver observations in the presence of strong out-of-band (OOB) interference, front-end saturation nonlinearities, additive receiver noise, and finite-resolution quantization. LMLVAMP integrates spectral priors with neural network–based denoising within a principled message-passing algorithmic structure. Applications include upper mid-band (FR3, 7–24 GHz) wideband radio receivers for 6G, where spectrally separated interference and hardware-induced nonlinear distortions degrade performance beyond the capacity of conventional linear methods (Joy et al., 30 Jan 2026).

1. System and Observation Model

The system considers $N$-length time-domain samples $r \in \mathbb{C}^N$ comprising a superposition of $L$ frequency-domain sources:

$r = \sum_{\ell=0}^{L-1} V x_\ell,$

where $V$ is the $N \times N$ unitary inverse discrete Fourier transform (IDFT), and each $x_\ell[k]$ is nonzero only within its designated frequency band $B_\ell$, with $B_\ell \cap B_k = \emptyset$ for $\ell \ne k$.

The frequency-domain coefficients $x_\ell$ are assigned Gaussian priors:

$$x_\ell[i] \sim \begin{cases} \mathcal{N}(\mu_\ell[i], S_\ell) & i \in B_\ell, \ 0 & \text{otherwise.} \end{cases}$$

In typical use, $\ell=0$ denotes the desired user, and $\ell=1$ indicates a spectrally separated interferer.

The receiver front-end applies a smooth, memoryless amplitude-compressing nonlinearity, followed by additive white Gaussian noise and, optionally, scalar quantization:

$$y[i] = \phi(r[i], w_a[i], w_b[i]) = f\left( \frac{|r[i] + w_a[i]|}{\sqrt{P_{\rm sat}}} \right)(r[i] + w_a[i]) + w_b[i],$$

with

$f(x) = \frac{\tanh(x)}{x},$

$w_a[i] \sim \mathcal{CN}(0, \sigma_a^2)$ (pre-nonlinearity noise), $w_b[i] \sim \mathcal{CN}(0, \sigma_b^2)$ (post-nonlinearity noise), and $P_{\rm sat}$ the saturation threshold. For finite-resolution analog-to-digital conversion (ADC), a uniform scalar quantizer $\mathcal{Q}$ is applied: $y_q[i] = \mathcal{Q}(y[i])$.

2. Multi-Layer VAMP Algorithmic Structure

Signal recovery is formulated as inference in a two-layer network:

• Layer 0 (Spectral): $x \mapsto r = Vx$
• Layer 1 (Nonlinear): $r \mapsto y = \Phi(r, w)$

The classical Multi-Layer Vector Approximate Message Passing (ML-VAMP) algorithm alternates two denoising steps per iteration, transitioning between frequency and time domains via orthogonal transforms. The denoisers are augmented by Onsager-like corrections for improved convergence. The Bayesian ML-VAMP update sequence is: $$\begin{aligned} x^{(t)} &= G_0(z_0^{(t)}, \gamma_0^{(t)}), \quad \alpha_0^{(t)} = \langle G_0' \rangle, \ z_1^{(t)} &= \frac{V(x^{(t)} - \alpha_0^{(t)} z_0^{(t)})}{1 - \alpha_0^{(t)}}, \quad \gamma_1^{(t)} = \gamma_0^{(t)}(1/\alpha_0^{(t)} - 1), \ r^{(t)} &= G_1(z_1^{(t)}, \gamma_1^{(t)}), \quad \alpha_1^{(t)} = \langle G_1' \rangle, \ z_0^{(t+1)} &= \frac{V(r^{(t)} - \alpha_1^{(t)} z_1^{(t)})}{1 - \alpha_1^{(t)}}, \quad \gamma_0^{(t+1)} = \gamma_1^{(t)}(1/\alpha_1^{(t)} - 1). \end{aligned}$$ Here, $G_0$ is the spectral (linear-Gaussian) denoiser, yielding, for each frequency bin $i$:

$$x[i] = \begin{cases} \mu_\ell[i] + \frac{\gamma_0 S_\ell}{1+\gamma_0 S_\ell}(z_0[i] - \mu_\ell[i]) & i \in B_\ell, \ 0 & i \notin \cup_\ell B_\ell, \end{cases}$$

with average divergence $$\alpha_0 = \sum_\ell \frac{|B_\ell|}{N} \frac{\gamma_0 S_\ell}{1+\gamma_0 S_\ell}$$. The nonlinear denoiser $G_1$ is the componentwise conditional mean estimator $$\mathbb{E}\{ r \mid y = \phi(r, w), r \sim \mathcal{CN}(z_1, \gamma_1^{-1}) \}$$, which lacks a closed-form in presence of saturation and quantization.

3. Learned Neural Network Denoisers

LMLVAMP generalizes ML-VAMP by replacing analytic denoisers $G_0$ and $G_1$ with small, trainable neural network denoisers $f_0$ and $f_1$. These networks learn to emulate minimum mean-squared error (MMSE) properties and estimate Onsager-correcting divergences.

• Spectral Message Updater ($f_0$): For each frequency bin $i$ at iteration $t$,

$$\rho_0^{(t)}[i] = f_0(z_0^{(t)}[i], \gamma_0^{(t)}, S[i], \mu[i], t;\theta) \to (\tilde\beta_0[i], \tilde\beta_1[i]),$$

with input features $$[|z_0|/\sqrt{P_{\rm sat}}, (\gamma_0 P_{\rm sat})^{-1}, S/P_{\rm sat}, |\mu|/\sqrt{P_{\rm sat}}]$$. The iteration-wide coefficients are

$$(\beta_0^{(t)}, \beta_1^{(t)}) = \frac{1}{N} \sum_{i=0}^{N-1} \rho_0^{(t)}[i].$$

• Nonlinear Denoiser ($f_1$): For each sample $i$,

$$(v^{(t)}[i], \rho_1^{(t)}[i]) = f_1(z_1^{(t)}[i], \gamma_1^{(t)}, y[i], t;\theta),$$

with features $$[|z_1|/\sqrt{P_{\rm sat}}, (\gamma_1 P_{\rm sat})^{-1}, |y|/\sqrt{P_{\rm sat}}]$$. Both $f_0$ and $f_1$ are two-layer networks (64 sigmoid units with linear outputs).

4. Algorithmic Workflow and Pseudocode

The LMLVAMP inference procedure unrolls for $T$ iterations as follows:

1. Initialization: $z_1^{(0)} \leftarrow V \mu$, $\gamma_1^{(0)} \leftarrow S^{-1}$.
2. For $t=0,...,T-1$:
   • Nonlinear denoising: For all $i$, $$(v[i], \rho_1[i]) \leftarrow f_1(z_1[i], \gamma_1, y[i], t; \theta)$$.
   • Spectral transformation: $z_0 \leftarrow V v$, $\gamma_0 \leftarrow (1/N) \sum_i \rho_1[i]$.
   • Spectral denoising: $$x \leftarrow \mu + \left(\frac{\gamma_0 S}{1+\gamma_0 S}\right) \odot (z_0 - \mu)$$.
   • Update: $$\gamma_1 \leftarrow \gamma_0 \cdot \left(\frac{1}{N} \sum_i \frac{\gamma_0 S[i]}{1+\gamma_0 S[i]}\right)^{-1}$$.
   • Message update: For all $i$, $$\rho_0[i] \leftarrow f_0(z_0[i], \gamma_0, S[i], \mu[i], t;\theta)$$.
   • Aggregate: $$(\beta_0, \beta_1) \leftarrow (1/N) \sum_i \rho_0[i]$$.
   • Next iterate: $z_1 \leftarrow \beta_0 V x - \beta_1 V z_0$.
3. User-band selection: $\hat{x}_0 \leftarrow m_{B_0} \odot x$.

Forward and inverse FFT operations are leveraged for computational efficiency ($O(N)$ per iteration).

5. Training Objectives and Optimization

Trainable parameters $\theta$ of $f_0$ and $f_1$ are optimized end-to-end via backpropagation through the $T$-step unrolled LMLVAMP. The total loss function is a convex combination of:

• Early-iteration loss:

$$\mathcal{L}_{\rm early} = \sum_{t=1}^{T-1} w_t \| x_0[B_0] - \hat{x}^{(t)}[B_0] \|^2, \qquad w_t = \frac{t}{\sum_{i=1}^{T-1} i}$$

• Final-iteration loss: $$\mathcal{L}_{\rm final} = \| x_0[B_0] - \hat{x}^{(T)}[B_0] \|^2$$
• Total loss: $$\mathcal{L}_{\rm total} = \eta \mathcal{L}_{\rm final} + (1-\eta) \mathcal{L}_{\rm early}$$ with $\eta \in (0.5, 1]$

Optimization employs Adam with exponential learning-rate decay. Regularization is enforced by restricting network size and introducing weighted intermediate losses for training stability.

6. Performance and Evaluation in FR3 Coexistence Scenarios

Simulated configurations included $N=512$, $B_0 = [0, 100)$, $B_1 = [300, 400)$, pre-nonlinearity noise $\sigma_a^2 = 0$ dB, post-nonlinearity noise $\sigma_b^2 = -10$ dB, saturation SNR $=40$ dB, signal SNR $\in \{10, 20\}$ dB, interference-to-noise ratio (INR) $\in [30, 80]$ dB, and 10-bit quantization with 12 dB backoff. Competing estimators were LMLVAMP-K/U (with/without known interferer band), linear Wiener baselines, and an "oracle" ideal nonlinearity-inversion bound.

Metrics:

• Achievable rate lower bound: $C \geq -\log_2(1-\rho)$ (correlation coefficient $\rho$)
• Normalized MSE

Key results:

• LMLVAMP-K approached oracle rates within two iterations at INR $=80$ dB.
• LMLVAMP-U achieved $>$20 dB NMSE reduction compared to linear methods in saturation-dominated scenarios.
• With 10-bit quantization, LMLVAMP retained a $>$10 dB advantage over linear approaches.
• Additional algorithmic unfolding (iterations) yielded incremental performance improvements.
• Ablations fixing $(\beta_0, \beta_1) = (1, 0)$ had negligible performance loss, indicating stable convergence.

LMLVAMP's O($N$) per-iteration complexity and compact parameterization (small neural nets) demonstrate scalability to large systems and robust gains in realistic 6G coexistence conditions (Joy et al., 30 Jan 2026).

7. Significance and Implications

LMLVAMP exemplifies the hybridization of model-based inference (structured priors, orthogonal transforms) and machine-learned components (data-driven denoisers) within message-passing frameworks. By incorporating spectral priors and leveraging neural denoisers to bypass intractable conditional expectations due to nonlinearities and quantization, LMLVAMP successfully bridges practical front-end hardware constraints and advanced signal recovery objectives. This architecture is particularly suitable for wideband communications in spectrally dense environments, such as future FR3 6G scenarios, where classical linear estimators fail to address nonlinear spectral leakage and quantization artifacts.

A plausible implication is the extensibility of LMLVAMP principles to other nonlinear or quantized inference tasks in communications and signal processing, given its stability, computational efficiency, and end-to-end trainability.