Attention-based cross-modality interaction for multispectral pedestrian detection

Tianshan Liu, Rui Zhao, Kin Man Lam

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

1 Citation (Scopus)


Multispectral pedestrian detection has attracted extensive attention, as paired RGB-thermal images can provide complementary patterns to deal with illumination changes in realistic scenarios. However, most of the existing deep-learning-based multispectral detectors extract features from RGB and thermal inputs separately, and fuse them by a simple concatenation operation. This fusion strategy is suboptimal, as undifferentiated concatenation for each region and feature channel may hamper the optimal selection of complementary features from different modalities. To address this limitation, in this paper, we propose an attention-based cross-modality interaction (ACI) module, which aims to adaptively highlight and aggregate the discriminative regions and channels of the feature maps from RGB and thermal images. The proposed ACI module is deployed into multiple layers of a two-branch-based deep architecture, to capture the cross-modal interactions from diverse semantic levels, for illumination-invariant pedestrian detection. Experimental results on the public KAIST multispectral pedestrian benchmark show that the proposed method achieves state-of-the-art detection performance.

Original languageEnglish
Title of host publicationInternational Workshop on Advanced Imaging Technology, IWAIT 2021
EditorsMasayuki Nakajima, Jae-Gon Kim, Wen-Nung Lie, Qian Kemao
ISBN (Electronic)9781510643642
Publication statusPublished - Mar 2021
Event2021 International Workshop on Advanced Imaging Technology, IWAIT 2021 - Kagoshima, Virtual, Japan
Duration: 5 Jan 20216 Jan 2021

Publication series

NameProceedings of SPIE - The International Society for Optical Engineering
ISSN (Print)0277-786X
ISSN (Electronic)1996-756X


Conference2021 International Workshop on Advanced Imaging Technology, IWAIT 2021
CityKagoshima, Virtual

ASJC Scopus subject areas

  • Electronic, Optical and Magnetic Materials
  • Condensed Matter Physics
  • Computer Science Applications
  • Applied Mathematics
  • Electrical and Electronic Engineering


Dive into the research topics of 'Attention-based cross-modality interaction for multispectral pedestrian detection'. Together they form a unique fingerprint.

Cite this