FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion

Yuxiang Sun, Weixun Zuo, Peng Yun, Hengli Wang, Ming Liu

Research output: Journal article publicationJournal articleAcademic researchpeer-review

87 Citations (Scopus)


Semantic segmentation of urban scenes is an essential component in various applications of autonomous driving. It makes great progress with the rise of deep learning technologies. Most of the current semantic segmentation networks use single-modal sensory data, which are usually the RGB images produced by visible cameras. However, the segmentation performance of these networks is prone to be degraded when lighting conditions are not satisfied, such as dim light or darkness. We find that thermal images produced by thermal imaging cameras are robust to challenging lighting conditions. Therefore, in this article, we propose a novel RGB and thermal data fusion network named FuseSeg to achieve superior performance of semantic segmentation in urban scenes. The experimental results demonstrate that our network outperforms the state-of-the-art networks. Note to Practitioners - This article investigates the problem of semantic segmentation of urban scenes when lighting conditions are not satisfied. We provide a solution to this problem via information fusion with RGB and thermal data. We build an end-to-end deep neural network, which takes as input a pair of RGB and thermal images and outputs pixel-wise semantic labels. Our network could be used for urban scene understanding, which serves as a fundamental component of many autonomous driving tasks, such as environment modeling, obstacle avoidance, motion prediction, and planning. Moreover, the simple design of our network allows it to be easily implemented using various deep learning frameworks, which facilitates the applications on different hardware or software platforms.

Original languageEnglish
Article number9108585
Pages (from-to)1000-1011
Number of pages12
JournalIEEE Transactions on Automation Science and Engineering
Issue number3
Publication statusPublished - Jul 2021
Externally publishedYes


  • Autonomous driving
  • information fusion
  • semantic segmentation
  • thermal images
  • urban scenes

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering


Dive into the research topics of 'FuseSeg: Semantic Segmentation of Urban Scenes Based on RGB and Thermal Data Fusion'. Together they form a unique fingerprint.

Cite this