Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks

Kangzhong Wang; M. K.Michael Cheung; Youqian Zhang; Chunxi Yang; Peter Q. Chen; Eugene Yujun Fu; Grace Ngai

doi:10.1145/3581783.3612870

Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks

Kangzhong Wang, M. K.Michael Cheung, Youqian Zhang, Chunxi Yang, Peter Q. Chen, Eugene Yujun Fu, Grace Ngai

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

1 Citation (Scopus)

Abstract

Automatic detection of backchannel has great potential to enhance artificial mediators, which indicate listeners' attention and agreement in human communication. It is often expressed by subtle non-verbal cues that occur briefly and sparsely. Focusing on identifying and locating these subtle cues (i.e., their occurrence moment and the involved body parts), this paper proposes a novel approach for backchannel detection. In particular, our model utilizes temporal- and modality-attention modules to determine and lead the model to pay more attention to both the indicative moment and the accompanying body parts at that specific time. It achieves an accuracy of 68.6% on the testing set in MultiMediate'23 backchannel detection challenge, outperforming the counterparts. Furthermore, we conducted an ablation study to thoroughly understand the contributions of our model. This study underscores the effectiveness of our selection of modality inputs and the importance of the two attention modules in our model.

Original language	English
Title of host publication	MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia
Publisher	Association for Computing Machinery, Inc
Pages	9586-9590
Number of pages	5
ISBN (Electronic)	9798400701085
DOIs	https://doi.org/10.1145/3581783.3612870
Publication status	Published - 26 Oct 2023
Event	31st ACM International Conference on Multimedia, MM 2023 - Ottawa, Canada Duration: 29 Oct 2023 → 3 Nov 2023

Publication series

Name	MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

Conference

Conference	31st ACM International Conference on Multimedia, MM 2023
Country/Territory	Canada
City	Ottawa
Period	29/10/23 → 3/11/23

Keywords

attention models
backchannel detection
visual cues

ASJC Scopus subject areas

Artificial Intelligence
Computer Graphics and Computer-Aided Design
Human-Computer Interaction
Software

More information

10.1145/3581783.3612870

Cite this

Wang, K., Cheung, M. K. M., Zhang, Y., Yang, C., Chen, P. Q., Fu, E. Y., & Ngai, G. (2023). Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks. In MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia (pp. 9586-9590). (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia). Association for Computing Machinery, Inc. https://doi.org/10.1145/3581783.3612870

Wang, Kangzhong ; Cheung, M. K.Michael ; Zhang, Youqian et al. / Unveiling Subtle Cues : Backchannel Detection Using Temporal Multimodal Attention Networks. MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, Inc, 2023. pp. 9586-9590 (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia).

@inproceedings{18b5579f5ff747dbb08b0f7af53ebd7b,

title = "Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks",

abstract = "Automatic detection of backchannel has great potential to enhance artificial mediators, which indicate listeners' attention and agreement in human communication. It is often expressed by subtle non-verbal cues that occur briefly and sparsely. Focusing on identifying and locating these subtle cues (i.e., their occurrence moment and the involved body parts), this paper proposes a novel approach for backchannel detection. In particular, our model utilizes temporal- and modality-attention modules to determine and lead the model to pay more attention to both the indicative moment and the accompanying body parts at that specific time. It achieves an accuracy of 68.6% on the testing set in MultiMediate'23 backchannel detection challenge, outperforming the counterparts. Furthermore, we conducted an ablation study to thoroughly understand the contributions of our model. This study underscores the effectiveness of our selection of modality inputs and the importance of the two attention modules in our model.",

keywords = "attention models, backchannel detection, visual cues",

author = "Kangzhong Wang and Cheung, {M. K.Michael} and Youqian Zhang and Chunxi Yang and Chen, {Peter Q.} and Fu, {Eugene Yujun} and Grace Ngai",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 31st ACM International Conference on Multimedia, MM 2023 ; Conference date: 29-10-2023 Through 03-11-2023",

year = "2023",

month = oct,

day = "26",

doi = "10.1145/3581783.3612870",

language = "English",

series = "MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia",

publisher = "Association for Computing Machinery, Inc",

pages = "9586--9590",

booktitle = "MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia",

}

Wang, K, Cheung, MKM, Zhang, Y, Yang, C, Chen, PQ, Fu, EY & Ngai, G 2023, Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks. in MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia. MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia, Association for Computing Machinery, Inc, pp. 9586-9590, 31st ACM International Conference on Multimedia, MM 2023, Ottawa, Canada, 29/10/23. https://doi.org/10.1145/3581783.3612870

Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks. / Wang, Kangzhong; Cheung, M. K.Michael; Zhang, Youqian et al.
MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, Inc, 2023. p. 9586-9590 (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia).

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - Unveiling Subtle Cues

T2 - 31st ACM International Conference on Multimedia, MM 2023

AU - Wang, Kangzhong

AU - Cheung, M. K.Michael

AU - Zhang, Youqian

AU - Yang, Chunxi

AU - Chen, Peter Q.

AU - Fu, Eugene Yujun

AU - Ngai, Grace

PY - 2023/10/26

Y1 - 2023/10/26

N2 - Automatic detection of backchannel has great potential to enhance artificial mediators, which indicate listeners' attention and agreement in human communication. It is often expressed by subtle non-verbal cues that occur briefly and sparsely. Focusing on identifying and locating these subtle cues (i.e., their occurrence moment and the involved body parts), this paper proposes a novel approach for backchannel detection. In particular, our model utilizes temporal- and modality-attention modules to determine and lead the model to pay more attention to both the indicative moment and the accompanying body parts at that specific time. It achieves an accuracy of 68.6% on the testing set in MultiMediate'23 backchannel detection challenge, outperforming the counterparts. Furthermore, we conducted an ablation study to thoroughly understand the contributions of our model. This study underscores the effectiveness of our selection of modality inputs and the importance of the two attention modules in our model.

AB - Automatic detection of backchannel has great potential to enhance artificial mediators, which indicate listeners' attention and agreement in human communication. It is often expressed by subtle non-verbal cues that occur briefly and sparsely. Focusing on identifying and locating these subtle cues (i.e., their occurrence moment and the involved body parts), this paper proposes a novel approach for backchannel detection. In particular, our model utilizes temporal- and modality-attention modules to determine and lead the model to pay more attention to both the indicative moment and the accompanying body parts at that specific time. It achieves an accuracy of 68.6% on the testing set in MultiMediate'23 backchannel detection challenge, outperforming the counterparts. Furthermore, we conducted an ablation study to thoroughly understand the contributions of our model. This study underscores the effectiveness of our selection of modality inputs and the importance of the two attention modules in our model.

KW - attention models

KW - backchannel detection

KW - visual cues

UR - http://www.scopus.com/inward/record.url?scp=85179552850&partnerID=8YFLogxK

U2 - 10.1145/3581783.3612870

DO - 10.1145/3581783.3612870

M3 - Conference article published in proceeding or book

AN - SCOPUS:85179552850

T3 - MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

SP - 9586

EP - 9590

BT - MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia

PB - Association for Computing Machinery, Inc

Y2 - 29 October 2023 through 3 November 2023

ER -

Wang K, Cheung MKM, Zhang Y, Yang C, Chen PQ, Fu EY et al. Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks. In MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, Inc. 2023. p. 9586-9590. (MM 2023 - Proceedings of the 31st ACM International Conference on Multimedia). doi: 10.1145/3581783.3612870

Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

1st Place in the Backchannel Detection Challenge at ACM MM'23

Cite this

Unveiling Subtle Cues: Backchannel Detection Using Temporal Multimodal Attention Networks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

Prizes

1st Place in the Backchannel Detection Challenge at ACM MM'23

Cite this