Multimodal integration for data-driven classification of mental fatigue during construction equipment operations: Incorporating electroencephalography, electrodermal activity, and video signals

Imran Mehmood, Heng Li, Waleed Umer, Aamir Arsalan, Shahnawaz Anwer, Mohammed Aquil Mirza, Jie Ma, Maxwell Fordjour Antwi-Afari

Research output: Journal article publicationJournal articleAcademic researchpeer-review

9 Citations (Scopus)

Abstract

Construction equipment operations that require high levels of attention can cause mental fatigue, which can lead to inefficiencies and accidents. Previous studies classified mental fatigue using single-modal data with acceptable accuracy. However, mental fatigue is a multimodal problem, and no single modality is superior. Moreover, none of the previous studies in construction industry have investigated multimodal data fusion for classifying mental fatigue and whether such an approach would improve mental fatigue detection. This study proposes a novel approach using three machine learning models and multimodal data fusion to classify mental fatigue states. Electroencephalography, electrodermal activity, and video signals were acquired during an excavation operation, and the decision tree model using multimodal sensor data fusion outperformed other models with 96.2% accuracy and 96.175%–98.231% F1 scores. Multimodal sensor data fusion can aid in the development of a real-time system to classify mental fatigue and improve safety management at construction sites.

Original languageEnglish
Article number100198
JournalDevelopments in the Built Environment
Volume15
DOIs
Publication statusPublished - Oct 2023

Keywords

  • Construction equipment operators
  • Construction safety
  • Machine learning
  • Mental fatigue
  • Multimodal data

ASJC Scopus subject areas

  • Architecture
  • Civil and Structural Engineering
  • Building and Construction
  • Materials Science (miscellaneous)
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design

Fingerprint

Dive into the research topics of 'Multimodal integration for data-driven classification of mental fatigue during construction equipment operations: Incorporating electroencephalography, electrodermal activity, and video signals'. Together they form a unique fingerprint.

Cite this