A multimodal human-robot sign language interaction framework applied in social robots

Jie Li, Junpei Zhong, Ning Wang (Corresponding Author)

Research output: Journal article publicationJournal articleAcademic researchpeer-review

6 Citations (Scopus)

Abstract

Deaf-mutes face many difficulties in daily interactions with hearing people through spoken language. Sign language is an important way of expression and communication for deaf-mutes. Therefore, breaking the communication barrier between the deaf-mute and hearing communities is significant for facilitating their integration into society. To help them integrate into social life better, we propose a multimodal Chinese sign language (CSL) gesture interaction framework based on social robots. The CSL gesture information including both static and dynamic gestures is captured from two different modal sensors. A wearable Myo armband and a Leap Motion sensor are used to collect human arm surface electromyography (sEMG) signals and hand 3D vectors, respectively. Two modalities of gesture datasets are preprocessed and fused to improve the recognition accuracy and to reduce the processing time cost of the network before sending it to the classifier. Since the input datasets of the proposed framework are temporal sequence gestures, the long-short term memory recurrent neural network is used to classify these input sequences. Comparative experiments are performed on an NAO robot to test our method. Moreover, our method can effectively improve CSL gesture recognition accuracy, which has potential applications in a variety of gesture interaction scenarios not only in social robots.

Original languageEnglish
Article number1168888
JournalFrontiers in Neuroscience
Volume17
DOIs
Publication statusPublished - 11 Apr 2023

Keywords

  • gesture recognition
  • human-robot interaction
  • multimodal sensors
  • sign language
  • social robots

ASJC Scopus subject areas

  • General Neuroscience

Fingerprint

Dive into the research topics of 'A multimodal human-robot sign language interaction framework applied in social robots'. Together they form a unique fingerprint.

Cite this