A Visual-Based Gesture Prediction Framework Applied in Social Robots

Bixiao Wu, Junpei Zhong, Chenguang Yang (Corresponding Author)

Research output: Journal article publicationJournal articleAcademic researchpeer-review

4 Citations (Scopus)

Abstract

In daily life, people use their hands in various ways for most daily activities. There are many applications based on the position, direction, and joints of the hand, including gesture recognition, gesture prediction, robotics and so on. This paper proposes a gesture prediction system that uses hand joint coordinate features collected by the Leap Motion to predict dynamic hand gestures. The model is applied to the NAO robot to verify the effectiveness of the proposed method. First of all, in order to reduce jitter or jump generated in the process of data acquisition by the Leap Motion, the Kalman filter is applied to the original data. Then some new feature descriptors are introduced. The length feature, angle feature and angular velocity feature are extracted from the filtered data. These features are fed into the long-short time memory recurrent neural network (LSTM-RNN) with different combinations. Experimental results show that the combination of coordinate, length and angle features achieves the highest accuracy of 99.31%, and it can also run in real time. Finally, the trained model is applied to the NAO robot to play the finger-guessing game. Based on the predicted gesture, the NAO robot can respond in advance.

Original languageEnglish
Pages (from-to)510-519
Number of pages10
JournalIEEE/CAA Journal of Automatica Sinica
Volume9
Issue number3
DOIs
Publication statusPublished - 13 Sep 2021
Externally publishedYes

Keywords

  • Finger-guessing game
  • gesture prediction
  • human-robot interaction
  • long-short time memory recurrent neural network (LSTM-RNN)
  • social robot

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Information Systems
  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'A Visual-Based Gesture Prediction Framework Applied in Social Robots'. Together they form a unique fingerprint.

Cite this