Monitoring and alerting of crane operator fatigue using hybrid deep neural networks in the prefabricated products assembly process

Xiao Li, H. L. Chi, W. F. Zhang, Qiping Shen

Research output: Unpublished conference presentation (presented paper, abstract, poster)Conference presentation (not published in journal/proceeding/book)Academic researchpeer-review

5 Citations (Scopus)


Crane operators fatigue is one of the significant constraints should be monitored. Otherwise, it may lead to inefficient crane operations and safety issues. Recently, many deep neural networks have been developed for fatigue monitoring of vehicle drivers by processing the image or video data. However, the challenge is to distinguish the slight variations of facial features among still and motion frames (e.g., nodding and head tilt, yawning and talking). It can be exacerbated in the scenarios for crane operators due to their constant head moving to track the loads’ position and recurrent communication (talking) with crane banksman. In contrast to previous approaches, which models spatial information and traditional temporal information for sequential processing, this study proposes a hybrid model can not only extract the spatial features by customized convolutional neural networks (CNN) but also enrich the modeling dynamic motions in the temporal dimension through the deep bidirectional long short-term memory (DBLSTM). This hybrid model is trained and evaluated on the very popular dataset NTHU-DDD, and the results show that the proposed architecture achieves 93.6% overall accuracy and outperform the previous models in the literature.

Original languageEnglish
Number of pages8
Publication statusPublished - Jul 2019
Event36th International Symposium on Automation and Robotics in Construction, ISARC 2019 - Banff, Canada
Duration: 21 May 201924 May 2019


Conference36th International Symposium on Automation and Robotics in Construction, ISARC 2019


  • Crane operator
  • Deep learning
  • Fatigue monitoring and alerting
  • Prefabricated construction

ASJC Scopus subject areas

  • Artificial Intelligence
  • Building and Construction
  • Human-Computer Interaction

Cite this