Towards Proactive Human Robot Collaborative Assembly: A Multimodal Transfer Learning-Enabled Action Prediction Approach

Shufei Li, Pai Zheng, Junming Fan, Lihui Wang

Research output: Journal article publicationJournal articleAcademic researchpeer-review

Abstract

Human-robot collaborative assembly (HRCA) is vital for achieving high-level flexible automation for mass personalization in todays smart factories. However, existing works in both industry and academia mainly focus on adaptive robot planning, while seldom consider human operators intentions in advance. Hence, it hinders the HRCA transition towards a proactive manner. To overcome the bottleneck, this research proposes a multimodal transfer learning-enabled action prediction approach, serving as the prerequisite to ensure the Proactive HRCA. Firstly, a multimodal intelligence-based action recognition approach is proposed to predict ongoing human actions by leveraging the visual stream and skeleton stream with short-time input frames. Secondly, a transfer learning-enabled model is adapted to transfer learned knowledge from daily activities to industrial assembly operations rapidly for online operator intention analysis. Thirdly, a dynamic decision-making mechanism including the robotic decision and motion control is described to allow mobile robots to assist operators in a proactive manner. Lastly, an aircraft bracket assembly task is demonstrated in the lab environment, and the comparative study result shows that the proposed approach outperforms other state-of-the-art ones for efficient action prediction.

Original languageEnglish
JournalIEEE Transactions on Industrial Electronics
DOIs
Publication statusAccepted/In press - 2021

Keywords

  • action recognition
  • transfer learning
  • Human-robot collaboration
  • multimodal intelligence

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Cite this