FTN-16QAM in Coherent Optical Systems

• FTN-16QAM is a coherent optical communication format that transmits 16-QAM symbols faster than the Nyquist limit, leveraging controlled intersymbol interference.
• It uses partial response signaling and a phase-tracking DFE with turbo equalization to mitigate ISI and phase noise, ensuring robust signal recovery.
• Experimental results show a 0.9 dB power margin improvement over MB-PCS-64QAM, underscoring FTN-16QAM's enhanced spectral efficiency and performance.

Faster-than-Nyquist 16-QAM (FTN-16QAM) is a coherent optical communication format in which 16-ary quadrature amplitude modulation symbols are transmitted at an intentionally accelerated symbol rate, exceeding the conventional Nyquist limit for bandwidth efficiency. FTN-16QAM leverages partial response signaling and iterative turbo equalization to combat the resulting controlled intersymbol interference (ISI) when deployed in amplifier-less, short-reach coherent systems. A precise DSP architecture employing phase-tracking partial response decision feedback equalization (PT-PRDFE) and iterative sequence detection enables FTN-16QAM to deliver a measurable power margin advantage over traditional probabilistically shaped (PCS) QAM formats, such as MB-PCS-64QAM, under realistic constraints for short-distance, high-speed fiber links (Zou et al., 25 Jan 2026).

1. Partial Response Signaling and System Model

FTN-16QAM transmits symbols $s[n]\in\mathcal{S}_{16}$ at a contracted symbol interval $T = aT_0$ with $a < 1$, thereby increasing symbol rate beyond the Nyquist symbol period $T_0$. Each symbol is pulse-shaped by a root-raised-cosine (RRC) filter $g(t)$, imparting controlled ISI rather than attempting to fully invert the channel to a Kronecker delta impulse. The transmit-side shaping and the optical fiber channel combine to create a discrete-time channel model: $$y[n] = \sum_{k=0}^{2} h_k\, s[n-k]\, e^{j\theta_n} + v[n]$$ with ISI coefficients $h_0=1$, $h_1=2$, $h_2=1$ corresponding to the second-order partial-response polynomial $H(z) = (1 + z^{-1})^2$, phase noise $\theta_n$, and AWGN $v[n}$. The system thus deliberately leaves a controllable two-symbol memory in the channel for further digital processing.

2. Phase-Tracking Partial Response DFE (PT-PRDFE)

The FTN-16QAM receiver employs a PT-PRDFE for joint mitigation of ISI and phase noise. The receiver first applies a phase rotation to each sample using a recursively updated estimate: $\tilde{y}[n] = y[n] \exp(-j\hat{\phi}[n-1])$ where $\hat{\phi}[n-1]$ is updated via a proportional-integral phase-locked loop (PLL): $$\hat{\phi}[n] = \hat{\phi}[n-1] + K_p \phi_e[n] + K_i \sum_{i=0}^{n} \phi_e[i]$$ with the phase error metric for decision-directed operation given by

$$\phi_e[n] = \Im\{\tilde{y}[n]\, \overline{\hat{s}[n]}\}$$

where $\hat{s}[n] = \mathrm{decide}(\tilde{y}[n])$. This phase tracker is applied prior to main equalization, providing robust compensation for laser phase noise at typical linewidths (e.g., 100 kHz at 45 Gbaud).

The DFE structure is defined by: $$z[n] = \sum_{i=0}^{L-1} c_i\, \tilde{y}[n-i] - \sum_{j=1}^{B} b_j\, \hat{s}[n-j]$$ where $L$ is the feedforward tap length (chosen to ensure the residual channel is $\{1,2,1\}$), $B=2$ (PR order), and LMS adaptation governs tap updates: $c_i[n+1] = c_i[n] + \mu\, e[n]\, \tilde{y}^*[n-i]$

$b_j[n+1] = b_j[n] - \mu\, e[n]\, \hat{s}^*[n-j]$

with $e[n]=\hat{s}[n] - z[n]$ and small $\mu$ for stability.

3. Turbo Equalization Strategy

Residual ISI and colored noise at the PT-PRDFE output are further suppressed by a “post-whitening” FIR filter. The equalized symbols then enter a BCJR (or Viterbi Algorithm) sequence detector, operating on a trellis defined by the original 16QAM symbol alphabet and the convolved response of the PR-DFE and post-filter. The turbo equalization loop exchanges log-likelihood ratios (LLRs) between the BCJR detector and an FEC decoder (e.g., LDPC): $$L_E^D(x_k) = \ln \frac{ \sum_{s_{k-1}, s_k: x_k=1} \alpha_{k-1}(s_{k-1})\, \gamma_k(s_{k-1}, s_k) \beta_k(s_k) }{ \sum_{s_{k-1}, s_k: x_k=0} \alpha_{k-1}(s_{k-1})\, \gamma_k(s_{k-1}, s_k) \beta_k(s_k) } - L_A^D(x_k)$$ where $L_A^D(x_k)$ is the a-priori bit LLR for the $k$-th symbol from the FEC decoder. Typically, three to five turbo iterations suffice for convergence.

4. Performance Metrics and Comparative Analysis

Benchmark BER for uncoded $M$-QAM in AWGN is approximated as: $$\mathrm{BER} \approx \frac{4(\sqrt{M}-1)}{\sqrt{M}\log_2 M} Q\left(\sqrt{\frac{3\,\text{SNR}}{M-1}}\right)$$ Measured BER vs. received optical power for FTN-16QAM with the described architecture closely tracks this idealized curve, indicating effective mitigation of ISI and phase noise. Quantitatively, at the 7% HD-FEC threshold (BER $\approx 4 \times 10^{-3}$), FTN-16QAM with iterative turbo equalization demonstrates a power-margin advantage of $0.9\,\mathrm{dB}$ relative to MB-PCS-64QAM. The comparison is summarized as:

Format	BER = 4×10⁻³ Crossing	Power Margin (dB)
MB-PCS-64QAM	~4.9 dB	4.9
FTN-16QAM + turbo	~4.0 dB	4.0

This shift directly measures the improvement afforded by FTN signaling and advanced equalization (Zou et al., 25 Jan 2026).

5. Implementation Aspects and Practical Considerations

The receiver DSP complexity is characterized as follows: the phase rotator requires $2$ real multipliers per sample, the $L$-tap feedforward filter and $B$-tap feedback filter require $2L$ and $2B$ real multipliers, respectively, while the PLL loop filter introduces negligible overhead. With $L=5$ and $B=2$ (2nd-order PR-DFE), the total is $16$ real multipliers (equivalent to $8$ complex). Latency is dominated by the feedforward filter ($(L-1)/2$-symbol group delay) and one-symbol feedback in the DFE; PLL adds effectively zero delay due to decision-directed updates. Implementation assumes perfect frequency offset estimation (FOE) and chromatic dispersion (CD) compensation, and the experimental setup is single-polarization, neglecting polarization-mode dispersion.

6. Experimental Context and Significance

The PT-PRDFE–BCJR turbo architecture is experimentally validated for FTN-16QAM at 45 Gbaud over 40 km standard single-mode fiber in amplifier-less short-reach scenarios. The approach yields a measurable 0.9 dB launch-power-margin improvement over MB-PCS-64QAM with standard FEC under matched phase-noise and bandwidth conditions. This substantiates the practical viability of FTN-16QAM with advanced DSP as a means to increase spectral efficiency and system robustness in modern coherent communication links (Zou et al., 25 Jan 2026).