Segmentation of Overlapping Cytoplasm in Cervical Smear Images via Adaptive Shape Priors Extracted from Contour Fragments

Youyi Song, Jing Qin, Baiying Lei, Bin Sheng, Kup Sze Choi, Lei Zhu

Research output: Journal article publicationJournal articleAcademic researchpeer-review

8 Citations (Scopus)


We present a novel approach for segmenting overlapping cytoplasm of cells in cervical smear images by leveraging the adaptive shape priors extracted from cytoplasm's contour fragments and shape statistics. The main challenge of this task is that many occluded boundaries in cytoplasm clumps are extremely difficult to be identified and, sometimes, even visually indistinguishable. Given a clump where multiple cytoplasms overlap, our method starts by cutting its contour into a set of contour fragments. We then locate the corresponding contour fragments of each cytoplasm by a grouping process. For each cytoplasm, according to the grouped fragments and a set of known shape references, we construct its shape and, then, connect the fragments to form a closed contour as the segmentation result, which is explicitly constrained by the constructed shape. We further integrate the intensity and curvature information, which is complementary to the shape priors extracted from contour fragments, into our framework to improve the segmentation accuracy. We propose to iteratively conduct fragments grouping, shape constructing, and fragments connecting for progressively refining the shape priors and improving the segmentation results. We extensively evaluate the effectiveness of our method on two typical cervical smear datasets. The experimental results demonstrate that our approach is highly effective and consistently outperforms the state-of-the-art approaches. The proposed method is general enough to be applied to other similar microscopic image segmentation tasks, where heavily overlapped objects exist.

Original languageEnglish
Article number8709975
Pages (from-to)2849-2862
Number of pages14
JournalIEEE Transactions on Medical Imaging
Issue number12
Publication statusPublished - Dec 2019


  • automatic cervical cancer screening
  • contour fragments
  • fragments grouping and connecting
  • Overlapping cytoplasm segmentation

ASJC Scopus subject areas

  • Software
  • Radiological and Ultrasound Technology
  • Computer Science Applications
  • Electrical and Electronic Engineering

Cite this