Vision-based holistic scene understanding towards proactive human–robot collaboration

Junming Fan, Pai Zheng (Corresponding Author), Shufei Li

Research output: Journal article publicationReview articleAcademic researchpeer-review

40 Citations (Scopus)


Recently human–robot collaboration (HRC) has emerged as a promising paradigm for mass personalization in manufacturing owing to the potential to fully exploit the strength of human flexibility and robot precision. To achieve better collaboration, robots should be capable of holistically perceiving and parsing the information of a working scene in real-time to plan proactively and act accordingly. Although excessive attentions have been paid to human cognition in existing works of HRC, there is a lack of a holistic consideration of other crucial elements of a working scene, especially when taking a further step towards Proactive HRC. Aiming to fill the gap, this paper provides a systematic review of computer vision-based holistic scene understanding in HRC scenarios, which mainly takes into account the cognition of object, human, and environment along with visual reasoning to gather and compile visual information into semantic knowledge for subsequent robot decision-making and proactive collaboration. Finally, challenges and potential research directions that can be largely facilitated by enhanced holistic perception techniques are also discussed.

Original languageEnglish
Article number102304
JournalRobotics and Computer-Integrated Manufacturing
Publication statusPublished - Jun 2022


  • Computer vision
  • Deep learning
  • Holistic scene understanding
  • Human–robot collaboration
  • Smart manufacturing

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Mathematics(all)
  • Computer Science Applications
  • Industrial and Manufacturing Engineering


Dive into the research topics of 'Vision-based holistic scene understanding towards proactive human–robot collaboration'. Together they form a unique fingerprint.

Cite this