AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop

Chengxi Li, Pai Zheng (Corresponding Author), Shufei Li, Yatming Pang, Carman K.M. Lee

Research output: Journal article publicationJournal articleAcademic researchpeer-review

51 Citations (Scopus)


The teleoperation and coordination of multiple industrial robots play an important role in today's industrial internet-based collaborative manufacturing systems. The user-friendly teleoperation approach allows operators from different manufacturing domains to reduce redundant learning costs and intuitively control the robot in advance. Nevertheless, only a few preliminary works have been introduced very recently, let alone its effective implementation in the manufacturing scenarios. To address the gap, this research proposes a novel multi-robot collaborative manufacturing system with human-in-the-loop control by leveraging the cutting-edge augmented reality (AR) and digital twin (DT) techniques. In the proposed system, the DTs of industrial robots are firstly mapped to physical robots and visualize them in the AR glasses. Meanwhile, a multi-robot communication mechanism is designed and implemented, to synchronize the state of robots in the twin. Moreover, a reinforcement learning algorithm is integrated into the robot motion planning to replace the conventional kinematics-based robot movement with corresponding target positions. Finally, three interactive AR-assisted DT modes, including real-time motion control, planned motion control, and robot monitoring mode are generated, which can be readily switched by human operators. Two experimental studies are conducted on (1) a single robot with a commonly used peg-in-hole experiment, and (2) the motion planning of multi-robot collaborative tasks, respectively. From the experimental results, it can be found that the proposed system can well handle the multi-robot teleoperation tasks with high efficiency and owns great potentials to be adopted in other complicated manufacturing scenarios in the near future.

Original languageEnglish
Article number102321
JournalRobotics and Computer-Integrated Manufacturing
Publication statusPublished - Aug 2022


  • Augmented reality
  • Collaborative manufacturing system
  • Digital twin
  • Human-in-the-loop control
  • Reinforcement learning

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Mathematics(all)
  • Computer Science Applications
  • Industrial and Manufacturing Engineering


Dive into the research topics of 'AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop'. Together they form a unique fingerprint.

Cite this