Towards Proactive Human Robot Collaborative Assembly: A Multimodal Transfer Learning-Enabled Action Prediction Approach

Shufei Li, Pai Zheng, Junming Fan, Lihui Wang

Research output: Journal article publicationJournal articleAcademic researchpeer-review

2 Citations (Scopus)


Human-robot collaborative assembly (HRCA) is vital for achieving high-level flexible automation for mass personalization in todays smart factories. However, existing works in both industry and academia mainly focus on adaptive robot planning, while seldom consider human operators intentions in advance. Hence, it hinders the HRCA transition towards a proactive manner. To overcome the bottleneck, this research proposes a multimodal transfer learning-enabled action prediction approach, serving as the prerequisite to ensure the Proactive HRCA. Firstly, a multimodal intelligence-based action recognition approach is proposed to predict ongoing human actions by leveraging the visual stream and skeleton stream with short-time input frames. Secondly, a transfer learning-enabled model is adapted to transfer learned knowledge from daily activities to industrial assembly operations rapidly for online operator intention analysis. Thirdly, a dynamic decision-making mechanism including the robotic decision and motion control is described to allow mobile robots to assist operators in a proactive manner. Lastly, an aircraft bracket assembly task is demonstrated in the lab environment, and the comparative study result shows that the proposed approach outperforms other state-of-the-art ones for efficient action prediction.

Original languageEnglish
Pages (from-to)8579-8588
JournalIEEE Transactions on Industrial Electronics
Issue number8
Publication statusAccepted/In press - 2021


  • action recognition
  • transfer learning
  • Human-robot collaboration
  • multimodal intelligence

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Cite this