Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment

Xueying Shi, Yueming Jin, Qi Dou, Jing Qin, Pheng Ann Heng

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

Abstract

Automated surgical gesture recognition is of great importance in robot-assisted minimally invasive surgery. However, existing methods assume that training and testing data are from the same domain, which suffers from severe performance degradation when a domain gap exists, such as the simulator and real robot. In this paper, we propose a novel unsupervised domain adaptation framework which can simultaneously transfer multi-modality knowledge, i.e., both kinematic and visual data, from simulator to real robot. It remedies the domain gap with enhanced transferable features by using temporal cues in videos, and inherent correlations in multi-modal towards recognizing gesture. Specifically, we first propose a Motion Direction Oriented Kinematics feature alignment (MDO-K) to align kinematics, which exploits temporal continuity to transfer motion directions with smaller gap rather than position values, relieving the adaptation burden. Moreover, we propose a Kinematic and Visual Relation Attention (KV-Relation-ATT) to transfer the co-occurrence signals of kinematics and vision. Such features attended by correlation similarity are more informative for enhancing domain-irreverent of the model. Two feature alignment strategies benefit the model mutually during the end-to-end learning process. We extensively evaluate our method for gesture recognition using DESK dataset with peg transfer procedure. Results show that our approach recovers the performance with great improvement gains, up to 12.91% in Accuracy and 20.16% in F1score without using any annotations in real robot.

Original languageEnglish
Title of host publicationIEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages9453-9460
Number of pages8
ISBN (Electronic)9781665417143
DOIs
Publication statusPublished - Sep 2021
Event2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021 - Prague, Czech Republic
Duration: 27 Sep 20211 Oct 2021

Publication series

NameIEEE International Conference on Intelligent Robots and Systems
ISSN (Print)2153-0858
ISSN (Electronic)2153-0866

Conference

Conference2021 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2021
Country/TerritoryCzech Republic
CityPrague
Period27/09/211/10/21

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint

Dive into the research topics of 'Domain Adaptive Robotic Gesture Recognition with Unsupervised Kinematic-Visual Data Alignment'. Together they form a unique fingerprint.

Cite this