Learning-Based Signal Recovery in Nonlinear Systems with Spectrally Separated Interference

Abstract: Upper Mid-Band (FR3, 7-24 GHz) receivers for 6G must operate over wide bandwidths in dense spectral environments, making them particularly vulnerable to strong adjacent-band interference and front-end nonlinearities. While conventional linear receivers can suppress spectrally separated interferers under ideal hardware assumptions, receiver saturation and finite-resolution quantization cause nonlinear spectral leakage that severely degrades performance in practical wideband radios. We study the recovery of a desired signal from nonlinear receiver observations corrupted by a high-power out-of-band interferer. The receiver front-end is modeled as a smooth, memoryless nonlinearity followed by additive noise and optional quantization. To mitigate these nonlinear and quantization-induced distortions, we propose a learned multi-layer Vector Approximate Message Passing (LMLVAMP) algorithm that incorporates spectral priors with neural network based denoising. Simulation results demonstrate significant performance gains over conventional methods, particularly in high-interference regimes representative of FR3 coexistence scenarios.