Constructing a Convolutional Neural Network with a Suitable Capacity for a Semantic Segmentation Task

Yalong Jiang, Zheru Chi

Research output: Chapter in book / Conference proceedingChapter in an edited book (as author)Academic researchpeer-review


Although the state-of-the-art performance has been achieved in many computer vision tasks such as image classification, object detection, saliency prediction and depth estimation, Convolutional Neural Networks (CNNs) still perform unsatisfactorily in some difficult tasks such as human parsing which is the focus of our research. The inappropriate capacity of a CNN model and insufficient training data both contribute to the failure in perceiving the semantic information of detailed regions. The feature representations learned by a high-capacity model cannot generalize to the variations in viewpoints, human poses and occlusions in real-world scenarios due to overfitting. On the other hand, the under-fitting problem prevents a low-capacity model from developing the representations which are sufficiently expressive. In this chapter, we propose an approach to estimate the complexity of a task and match the capacity of a CNN model to the complexity of a task while avoiding under-fitting and overfitting. Firstly, a novel training scheme is proposed to fully explore the potential of low-capacity CNN models. The scheme outperforms existing end-to-end training schemes and enables low-capacity models to outperform models with higher capacity. Secondly, three methods are proposed to optimize the capacity of a CNN model on a task. The first method is based on improving the orthogonality among kernels which contributes to higher computational efficiency and better performance. In the second method, the convolutional kernels within each layer are evaluated according to their semantic functions and contributions to the training and test accuracy. The kernels which only contribute to the training accuracy but has no effect on the testing accuracy are removed to avoid overfitting. In the third method, the capacity of a CNN model is optimized by adjusting the dependency among convolutional kernels. A novel structure of convolutional layers is proposed to reduce the number of parameters while maintaining the similar performance. Besides capacity optimization, we further propose a method to evaluate the complexity of a human parsing task. An independent CNN model is trained for this purpose using the labels for pose estimation. The evaluation on complexity is achieved based on estimated pose information in images. The proposed scheme for complexity evaluation was conducted on the Pascal Person Part dataset and the Look into Person dataset which are for human parsing. The schemes for capacity optimization were conducted on our models for human parsing which were trained on the two data sets. Both quantitative and qualitative results demonstrate that our proposed algorithms can match the capacity of a CNN model well to the complexity of a task.

Original languageEnglish
Title of host publicationStudies in Computational Intelligence
Number of pages32
ISBN (Electronic)978-3-030-31756-0
ISBN (Print)978-3-030-31755-3
Publication statusPublished - Oct 2019

Publication series

NameStudies in Computational Intelligence
ISSN (Print)1860-949X
ISSN (Electronic)1860-9503


  • Capacity optimization
  • Complexity evaluation
  • Convolutional neural networks (CNNs)
  • Over-fitting
  • Under-fitting

ASJC Scopus subject areas

  • Artificial Intelligence


Dive into the research topics of 'Constructing a Convolutional Neural Network with a Suitable Capacity for a Semantic Segmentation Task'. Together they form a unique fingerprint.

Cite this