Vision-based holistic scene understanding towards proactive human–robot collaboration

Recently human–robot collaboration (HRC) has emerged as a promising paradigm for mass personalization in manufacturing owing to the potential to fully exploit the strength of human flexibility and robot precision. To achieve better collaboration, robots should be capable of holistically perceiving and parsing the information of a working scene in real-time to plan proactively and act accordingly. Although excessive attentions have been paid to human cognition in existing works of HRC, there is a lack of a holistic consideration of other crucial elements of a working scene, especially when taking a further step towards Proactive HRC. Aiming to fill the gap, this paper provides a systematic review of computer vision-based holistic scene understanding in HRC scenarios, which mainly takes into account the cognition of object, human, and environment along with visual reasoning to gather and compile visual information into semantic knowledge for subsequent robot decision-making and proactive collaboration. Finally, challenges and potential research directions that can be largely facilitated by enhanced holistic perception techniques are also discussed.