Saturation-Aware Gain Shaping

• Saturation-aware gain shaping is a framework that integrates nonlinear saturation constraints into gain optimization to mitigate distortion and resource inefficiency.
• The approach tailors gain profiles across domains such as wireless MIMO, fiber amplifiers, and superconducting circuits, allowing systems to operate effectively near hardware limits.
• By employing analytical and iterative optimization methods, it achieves distortion cancellation, improved dynamic range, and efficient beamforming in large-scale systems.

A saturation-aware gain shaping objective is a mathematical optimization framework that explicitly incorporates the nonlinear, saturating behavior of amplification components or channel combining effects into the design of gain profiles, beamforming weights, or amplifier circuits. Saturation-aware gain shaping arises in diverse physical settings—from wireless MIMO precoding near power amplifier (PA) saturation, to fiber amplifier modal control under population inversion saturation, to wideband superconducting parametric amplifiers, and reconfigurable intelligent surfaces (RIS) with statistically shaped beamforming. The unifying theme is the reformulation of the conventional gain optimization to account for—and frequently exploit—the peculiarity that, as some hardware constraint (e.g., input power, number of array elements) increases, marginal gains saturate or even degrade due to nonlinear physics or statistical dependencies.

1. Foundation of Saturation-Aware Gain Shaping

The classical gain optimization objective in linear systems seeks to maximize array gain or output signal-to-noise ratio, subject to power or structural constraints, often neglecting nonlinear effects or limitations imposed by underlying physical or statistical processes. In practical high-power or large-scale scenarios, this linear paradigm fails: (1) Amplification devices such as PAs or parametric circuits exhibit reduced linearity and increased distortion as they approach saturation; (2) Optical fiber amplifiers enter a saturation regime where gain is limited by available inversion; (3) Spatially extended arrays operating under statistical CSI or strongly correlated scattering realize sublinear or saturating beamforming gain.

The saturation-aware gain shaping objective modifies the canonical optimization problem by embedding constraints, penalty terms, or model-dependent expressions that account for these saturating phenomena. This approach ensures robust, distortion-minimizing, and resource-efficient operation, often requiring only modest algorithmic complexity or knowledge of device characteristics.

2. Mathematical Structure in Large-Array Nonlinear Precoding

In multi-antenna wireless downlink with nonlinear PAs, the main challenge is that higher transmit efficiency is achieved by operating close to saturation, but this incurs severe third-order (cubic) distortion which, under conventional schemes (e.g., maximum ratio transmission, MRT), adds coherently at the intended receiver, irreversibly degrading signal fidelity.

The Z3RO (Zero Third-Order) gain-shaping precoder solves this by recasting the beamforming weight optimization as:

$$\underset{g_0,\dots,g_{M-1}\in\mathbb C}{\text{maximize}} \ \left| \sum_{m=0}^{M-1} h_m g_m \right|^2 \quad \text{subject to} \quad \sum_{m=0}^{M-1} |g_m|^2 = 1, \quad \sum_{m=0}^{M-1} h_m g_m^3 = 0$$

The cubic distortion cancellation constraint compels at least one antenna to carry a large, phase-inverted gain, purposefully “over-driving” (i.e., saturating) that PA to make its cubic distortion destructively interfere with the aggregate third-order distortion of other antennas at the user.

In closed form, for a pure line-of-sight setting with constant $|h_m|$:

• All but one gain set to $\alpha$,
• The “saturated” antenna set to $-\alpha (M-1)^{1/3}$,
• $\alpha = \left[(M-1)+(M-1)^{2/3}\right]^{-1/2}$
• The array-gain loss relative to MRT vanishes as $M \rightarrow \infty$, while distortion is directly nulled for any $M$ (Rottenberg et al., 2022, Rottenberg et al., 2021).

Thus, the saturation-aware objective here is precisely the joint maximization of array gain subject to a nonlinear “zero third-order” constraint—a departure from purely quadratic (linear) form.

3. Formulations in Fiber and Optical Gain Media

In high-power fiber amplifiers or multimode gain media, the population inversion saturates due to stimulated emission as the output signal grows, limiting achievable gain and introducing spatial non-uniformity and modal instabilities. In this context, the saturation-aware gain shaping objective is formulated as the maximization of the mode-instability threshold $P_\text{th}$, defined by the ratio of saturated laser gain $g_L(z)$ to the stimulated Rayleigh scattering gain $g_{\text{th}}(z)$:

$$J = \max_{\,G(r),\,L} P_\text{th} \ = \max_{\,G(r),\,L} \Bigl\{P_s(0):\min_{0\le z\le L}\frac{g_L(z;G,s)}{g_\text{th}(z;G,s)}\ge1\Bigr\}$$

subject to geometric, power, and Brillouin suppression constraints. Here, $G(r, z)$ is explicitly shaped—through doping profile, pump configuration, or fiber length optimization—to achieve high $P_\text{th}$ under saturation, enabling greater output while avoiding modal instabilities (Smith et al., 2013). Saturation is accounted for via local two-level system population dynamics, modifying both gain and index modulation coupling.

In the multimode case, spatially resolved, mode-coupled propagation equations for pump and signal fields are solved iteratively or via adjoint-based gradient methods, targeting desired transmission matrix properties under saturated, inhomogeneous gain (Sperber et al., 2019).

4. Circuit-Level Gain Shaping Under Compression

Josephson parametric amplifiers (JPAs) and related nonlinear devices impose strict limitations on gain profile, bandwidth, and dynamic range by their intrinsic saturation (1 dB compression point) as determined by the nonlinearity and critical current of superconducting elements.

The saturation-aware gain shaping problem here is formulated as an overview of, for example, a Chebyshev-matched, band-pass filter network:

$$\min  \int_{\Omega_1}^{\Omega_2} [G(\Omega; \{g_i\}) - G_0]^2 \; d\Omega$$

subject to

$$\max |G(\Omega) - G_0| \leq R_p,\quad BW \geq \Delta\omega_{req},\quad P_{sat}(\{g_i\}, I_c, N) \geq P_{min}$$

where $P_{sat}$ is a nonlinear function scaling as $\sim (I_c/N)^2$ and encodes the amplifier’s 1-dB-compression power. The optimization thus accounts for spectral flatness and rejection as well as the highly nonlinear saturation constraints determined by the device physics (Kaufman et al., 2023).

5. Saturation Dynamics in Spatially Structured Arrays and RIS

In large RIS-assisted wireless systems operating under a two-timescale model (statistical optimization of RIS phases, instantaneous base station combining), the cumulative beamforming gain $G(N)$ (with $N$ elements) saturates due to spatial correlation dictated by the power angular spectrum (PAS) of the channel:

$G(N) = \max_{\psi : |\psi_m|=1} \psi^H C \psi$

with

$\lim_{N\to\infty} G(N) = G_{\max} < \infty$

where $C$ is a correlation matrix induced by the PAS. The saturation-aware gain-shaping objective is thus not simply to maximize $G(N)$, but to maximize

$$J(\psi) = G(N;\psi) - \lambda \left[\frac{G(N;\psi)}{G_\text{max}}\right]^p$$

or, alternately,

$$J(\psi) = G(N;\psi) \exp\left(-\alpha\, \frac{G(N;\psi)}{G_\text{max}}\right)$$

which penalizes the deployment of excess array elements with negligible marginal gain. This explicitly encodes the rapid diminishing returns present in such spatially correlated environments and guides RIS design towards efficient operation (Sadeghian et al., 29 Jul 2025).

6. Optimization Algorithms and Computational Aspects

Across domains, the saturation-aware gain shaping objective introduces nonconvexity and higher-order constraints compared to standard quadratic or convex gain maximization. Solution strategies include:

• Analytical closed-form solutions (as in Z3RO precoding for LoS arrays) (Rottenberg et al., 2022)
• Projected gradient or manifold ascent (RIS, e.g., to optimize $J(\psi)$ under unit-modulus constraints) (Sadeghian et al., 29 Jul 2025)
• Adjoint-mode or backpropagation-based gradient computation (fiber amplifier pump shaping) (Sperber et al., 2019)
• Line search for dual parameters (e.g., Lagrange multipliers in nonlinear constraints)

Computational complexity typically scales with system size or number of variables (antennas, modes, frequency bins), but remains tractable due to either analytic reduction or efficient iterative schemes. Global optimality is rare; local optima or probabilistic multistart strategies are applied.

7. Performance Trade-offs and Practical Implications

Saturation-aware gain shaping consistently trades reduced raw “linear” gain for substantial qualitative benefits:

• Distortion nulling: Z3RO suppresses coherent third-order distortion at the user, enabling high PA efficiency at the cost of a vanishing (as $M\to\infty$) reduction in array gain.
• Mode instability resilience: Shaped gain profiles in fiber amplifiers substantially raise instability thresholds beyond what would be obtained with naive, spatially uniform inversion.
• Bandwidth/dynamic range engineering: Circuit design incorporating gain profile shaping under compression extends usable dynamic range without violating gain flatness.
• Resource efficiency in RIS: Explicit penalization of over-deployment of array elements prevents waste when marginal beamforming gain saturates due to spatial correlation.

A plausible implication is that as system scale increases, saturation-aware design becomes not only beneficial but essential: neglecting saturation phenomena results in significant performance loss, distortion, or unproductive resource allocation.

Summary Table: Saturation-Aware Gain Shaping in Representative Domains

Domain	Objective Structure Examples	Principal Constraints/Effects
Large-array wireless	Max $|\sum h_m g_m|^2$, s.t. $\sum h_m g_m^3=0$, $\sum|g_m|^2=1$	PA cubic distortion, per-antenna saturation (Rottenberg et al., 2022)
Fiber amplifiers	Max $P_\text{th}$, s.t. $R(z)\ge1$ $\forall z$, pump/length constraints	Population inversion saturation, mode instability (Smith et al., 2013)
Josephson amplifiers	Min $\int (G(\Omega)-G_0)^2 d\Omega$, s.t. ripple, $P_{sat}\ge P_{min}$	Kerr/compression limits, Chebyshev gain profile (Kaufman et al., 2023)
RIS beamforming	Max $G(N)$, penalty for $G(N)\to G_{max}$	PAS-induced gain saturation, unit-modulus phases (Sadeghian et al., 29 Jul 2025)

Saturation-aware gain shaping thus provides a unifying optimization perspective for a broad class of technologically critical problems where nonlinearities and physical constraints set hard limits on achievable gain or array-based combining.