Sample-Based Neural Network Pre-Distorter for Transceiver Nonlinearity Compensation in IM/DD System

Zhiwei Chen, Wei Wang, Dongdong Zou, Weihao Ni, Mingzhu Yin, Xiaoliang Chen, Chao Lu, Dongmei Huang, Fan Li

Research output: Journal article publicationJournal articleAcademic researchpeer-review


Look-up table (LUT) enabled digital pre-distortion (DPD) is an effective means of nonlinear compensation. However, considering the storage requirement and feasibility of training process, the memory length is commonly limited to 3 or 5, which limits the nonlinear compensation performance of LUT-DPD. With the growing demands of high baud rate transmission, the system is more sensitive to nonlinear impairments, and LUT-DPD with short memory length is insufficient to meet the requirement on nonlinear compensation performance. Neural network (NN) has been proposed as a pre-distorter, while it requires precise modeling of the transceiver or complex multiplication operation. Following the training process of LUT, NN-based pre-distorter can also be trained on received samples, which is simpler to be implemented. Besides, giving scope to the learning ability of NN, a small number of samples are enough for training process. Moreover, the memory length can be easily expanded in NN-based method. Therefore, in this paper, we propose a sample-based NN-DPD which extends the memory length to 9 and even 13, with relatively low complexity. We experimentally demonstrate a transmission of beyond 100 Gbit/s PAM-6/PAM-8 signal in an intensity modulation and direct detection (IM/DD) system. For 40 Gbaud PAM-6 signal, a maximum receiver sensitivity gain of 1.5 dB is obtained at the KP4-FEC threshold when memory length is increased from 3 to 13. Compared to Volterra nonlinear equalizer (VNLE), the computational complexity of NN-DPD is reduced by nearly 60% when memory length is 9. The NN-DPD is a promising solution to reducing the influence of nonlinearity for future high-order modulation signal.

Original languageEnglish
Article number10343103
Pages (from-to)1-8
Number of pages8
JournalJournal of Lightwave Technology
Publication statusPublished - Dec 2023


  • Computational complexity
  • Convolutional neural networks
  • digital pre-distortion
  • look-up table
  • memory length
  • Modulation
  • Neural network
  • Nonlinear distortion
  • sample-based method
  • Symbols
  • Table lookup
  • Training
  • transceiver nonlinearity
  • Volterra nonlinear equalizer

ASJC Scopus subject areas

  • Atomic and Molecular Physics, and Optics


Dive into the research topics of 'Sample-Based Neural Network Pre-Distorter for Transceiver Nonlinearity Compensation in IM/DD System'. Together they form a unique fingerprint.

Cite this