TY - JOUR
T1 - RTFNet: RGB-Thermal Fusion Network for Semantic Segmentation of Urban Scenes
AU - Sun, Yuxiang
AU - Zuo, Weixun
AU - Liu, Ming
N1 - Funding Information:
Manuscript received November 7, 2018; accepted February 25, 2019. Date of publication March 13, 2019; date of current version April 24, 2019. This letter was recommended for publication by Associate Editor Y. Joo and Editor Y. Choi upon evaluation of the reviewers’ comments. This work was supported in part by the Shenzhen Science, Technology, and Innovation Commission (SZSTI) project JCYJ20160401100022706, in part by the National Natural Science Foundation of China project U1713211, in part by the Hong Kong University of Science and Technology Project IGN16EG12, and in part by the Hong Kong Research Grant Council (RGC) projects 11210017 and 21202816. (Corresponding author: Ming Liu.) The authors are with the Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Kowloon, Hong Kong SAR, China (e-mail: [email protected], [email protected]; [email protected]; [email protected]). Digital Object Identifier 10.1109/LRA.2019.2904733
Publisher Copyright:
© 2016 IEEE.
PY - 2019/7
Y1 - 2019/7
N2 - Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder-decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.
AB - Semantic segmentation is a fundamental capability for autonomous vehicles. With the advancements of deep learning technologies, many effective semantic segmentation networks have been proposed in recent years. However, most of them are designed using RGB images from visible cameras. The quality of RGB images is prone to be degraded under unsatisfied lighting conditions, such as darkness and glares of oncoming headlights, which imposes critical challenges for the networks that use only RGB images. Different from visible cameras, thermal imaging cameras generate images using thermal radiations. They are able to see under various lighting conditions. In order to enable robust and accurate semantic segmentation for autonomous vehicles, we take the advantage of thermal images and fuse both the RGB and thermal information in a novel deep neural network. The main innovation of this letter is the architecture of the proposed network. We adopt the encoder-decoder design concept. ResNet is employed for feature extraction and a new decoder is developed to restore the feature map resolution. The experimental results prove that our network outperforms the state of the arts.
KW - Deep Neural Network
KW - Information Fusion
KW - Semantic Segmentation
KW - Thermal Images
KW - Urban Scenes
UR - http://www.scopus.com/inward/record.url?scp=85064974110&partnerID=8YFLogxK
U2 - 10.1109/LRA.2019.2904733
DO - 10.1109/LRA.2019.2904733
M3 - Journal article
AN - SCOPUS:85064974110
SN - 2377-3766
VL - 4
SP - 2576
EP - 2583
JO - IEEE Robotics and Automation Letters
JF - IEEE Robotics and Automation Letters
IS - 3
M1 - 8666745
ER -