Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration

Weidong Huang, Leila Alem, Franco Tecchia, Henry Been Lirn Duh

Research output: Journal article publicationJournal articleAcademic researchpeer-review

53 Citations (Scopus)


Distributed collaborations between two or more participants on a task involving tangible artifacts (e.g., a machine, a patient, a tool) have become increasingly common in recent years due to rapid development in information and communication technologies. In this paper we focus on a specific type of remote-collaboration system where a remote helper guides a local worker using audio communication and hand gestures to perform a repair or a maintenance task. An established ICT approach to supporting this type of collaboration is to provide a shared visual space and some forms of remote gesture. The shared space typically consists of a video capture of the remote workspace which is then displayed on a 2D screen. However, this type of approach has its limitations. Firstly, it does not provide the helper with sufficient understanding of the spatial relationships between objects in the remote workspace. Secondly, it does not allow the helper to gesture in 3D. In an attempt to address these issues, we propose a Mixed Reality multimodal system that improves on previous 2D systems by introducing 3D real-time capturing and rendering of both the remote workspace and the helping hands and by creating a 3D shared visual space as a result of co-locating the remote workspace with the helping hands. In this system, we explore the possibility of increasing the feeling of immersion and co-presence by using head tracking, stereoscopic rendering, inter-occlusion handling and virtual shadowing. In this paper, we introduce HandsIn3D, a system that has been developed for the purpose of the proof of concept. We also present the results of experiments to verify the feasibility of our approach.

Original languageEnglish
Pages (from-to)77-89
Number of pages13
JournalJournal on Multimodal User Interfaces
Issue number2
Publication statusPublished - 1 Jun 2018
Externally publishedYes


  • 3D capture and rendering
  • Hand gesture
  • Microsoft Kinect
  • Mixed-reality
  • Remote collaboration
  • Shared visual space
  • Tele-presence

ASJC Scopus subject areas

  • Signal Processing
  • Human-Computer Interaction


Dive into the research topics of 'Augmented 3D hands: a gesture-based mixed reality system for distributed collaboration'. Together they form a unique fingerprint.

Cite this