A temporal spatial human digital twin approach for modeling human behavior with uncertainty

Modeling human behavior is paramount in the process of human-robot interaction (HRI). Human motion during HRI is uncertain. Even when the same individual performs the same action, it is impossible to ensure that the motion trajectory will be identical every time. These factors make modeling human behavior extremely challenging. Beyond human uncertainty, there are dynamic temporal spatial dependencies in HRI. Effectively capturing uncertainty while fully integrating temporal spatial features presents a significant challenge. Moreover, representing human behavior solely through human skeleton is insufficient. Recently, human digital twins have been developed to represent human geometry. However, current human digital twins are not well-suited for dynamic HRI scenarios, as they struggle to accurately depict high-dimensional human parameters, leading to issues such as nonlinear mapping and joint drift. In summary, existing methods find it difficult to address the human uncertainty, the temporal spatial dependencies, and the high-dimensional human parameters. To address the above challenges, this study proposes a temporal spatial human digital twin (TSHDT) for modeling human behavior in HRI. The TSHDT is based on predicted human skeletons and integrates forward and inverse kinematics along with diffusion prior distribution to represent high-dimensional human parameters, thus preventing joint drift and nonlinear mapping between joints. In developing the TSHDT, we introduce the human robot temporal spatial (HRTS) diffusion model to mitigate the uncertainty in human motion. The unique diffusion and denoising processes of the HRTS diffusion model can effectively submerge uncertainty in noise and accurately predict human motion during subsequent denoising steps. To ensure that the denoising process favors accuracy over diversity, we propose the temporal spatial fusion graph convolutional network (TSFGCN) to capture temporal spatial features between humans and robots, embedding them into the HRTS diffusion model. Finally, the effectiveness of the TSHDT was validated via predictive collision detection in human-robot fabric cutting experiments. Results demonstrate that the proposed method accurately models human behavior in collision detection experiments, achieving outstanding F1 scores.