Signal design and detection in presence of nonlinear phase noise

Pak Tao Lau, Joseph M. Kahn

Research output: Journal article publicationJournal articleAcademic researchpeer-review

100 Citations (Scopus)


In optical fiber transmission systems using inline amplifiers, the interaction of a signal and amplifier noise through the Kerr effect leads to nonlinear phase noise that can impair the detection of phase-modulated signals. We present analytical expressions for the maximum-likelihood (ML) decision boundaries and symbol-error rate (SER) for phase-shift keying and differential phase-shift keying systems with coherent and differentially coherent detection, respectively. The ML decision boundaries are in the form θ(r) = c2r2+ c1r + c0, where θ and r are the phase and the amplitude of the received signal, respectively. Using the expressions for the SER, we show that the impact of phase error from carrier synchronization is small, particularly for transoceanic links. For modulation formats such as 16-quadrature amplitude modulation, we propose various transmitter and receiver phase rotation strategies such that the ML detection is well approximated by using straight-line decision boundaries. The problem of signal constellation design for optimal SER performance is also studied for a system with four signal points.
Original languageEnglish
Pages (from-to)3008-3016
Number of pages9
JournalJournal of Lightwave Technology
Issue number10
Publication statusPublished - 1 Oct 2007
Externally publishedYes


  • Maximum likelihood (ML) detection
  • Nonlinear optics
  • Optical fiber communication
  • Optical Kerr effect
  • Phase noise
  • Quadrature amplitude modulation (QAM)

ASJC Scopus subject areas

  • Atomic and Molecular Physics, and Optics


Dive into the research topics of 'Signal design and detection in presence of nonlinear phase noise'. Together they form a unique fingerprint.

Cite this