Transformer-based Nonlinear Equalization for DP-16QAM Coherent Optical Communication Systems

Abstract: Compensating for nonlinear effects using digital signal processing (DSP) is complex and computationally expensive in long-haul optical communication systems due to intractable interactions between Kerr nonlinearity, chromatic dispersion (CD), and amplified spontaneous emission (ASE) noise from inline amplifiers. The application of machine learning architectures has demonstrated promising advancements in enhancing transmission performance through the mitigation of fiber nonlinear effects. In this paper, we apply a Transformer-based model to dual-polarisation (DP)-16QAM coherent optical communication systems. We test the performance of the proposed model for different values of fiber lengths and launched optical powers and show improved performance compared to the state-of-the-art digital backpropagation (DBP) algorithm, fully connected neural network (FCNN) and bidirectional long short term memory (BiLSTM) architecture.