Effects of dataset characteristics on the performance of fatigue detection for crane operators using hybrid deep neural networks

Fatigue of operators due to intensive workloads and long working time is a significant constraint that leads to inefficient crane operations and increased risk of safety issues. It can be potentially prevented through early warnings of fatigue for further appropriate work shift arrangements. Many deep neural networks have recently been developed for the fatigue detection of vehicle drivers through training and processing the facial image or video data from the public driver's datasets. However, these datasets are difficult to directly use for the fatigue detections under crane operation scenarios due to the variations of facial features and head movement patterns between crane operators and vehicle drivers. Furthermore, there is no representative and public dataset with the facial information of crane operators under construction scenarios. Therefore, this study aims to explore and analyse the features of multi-sources datasets and the corresponding data acquisition methods which are suitable for crane operators' fatigue detection, further providing collection guidelines of crane operators dataset. Variations on public datasets such as real or pretend facial expression, the segment level of human-verified labelling, camera positions, acquisition scenarios, and illumination conditions are analysed. A hybrid learning architecture is proposed by combining convolutional neural networks (CNN) and long short-term memory (LSTM) for fatigue detection. In order to establish a unified evaluation criterion, the effort of the study includes relabelling three public vehicle drivers datasets, NTHU-DDD, UTA-RLDD, and YawnDD, with human-verified labels at the frame and minute segment levels, and training the corresponding hybrid fatigue detection models accordingly. The average detection accuracies and losses are identified for the trained models of UTA-RLDD, NTHU-DDD, and YawnDD individually. The trained models are used to evaluate the fatigue status of facial videos from licensed crane operators under simulated crane operation scenarios. The results suggest the necessary considerations of different influential factors for establishing a large and public fatigue dataset for crane operators.