Fiber Nonlinearity Mitigation in Coherent Optical Systems

Abstract: Fiber nonlinearity represents a critical challenge to the capacity enhancement of modern optical communication systems. In recent years, significant research efforts have focused on mitigating its impact through two complementary approaches. On the one hand, researchers have investigated practical digital signal processing (DSP) techniques to mitigate or compensate for nonlinear impairments, such as reversing fiber propagation effects through digital backpropagation (DBP). However, the high computational complexity of these techniques often discourages their practical implementation. On the other hand, information-theoretic studies have sought to establish the capacity limits of the nonlinear optical fiber channel, providing a framework for evaluating the ultimate performance of existing optical networks and guiding the design of next-generation systems. This work reviews recent advances and proposes future directions for nonlinearity compensation and mitigation, including constellation shaping techniques and low-complexity DBP. Furthermore, it highlights the potential of these innovations both in advancing the theoretical understanding of fiber capacity limits and in enabling practical DSP implementations.