Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction

Ting Kong, Weili Fang, Peter E.D. Love, Hanbin Luo, Shuangjie Xu, Heng Li

Research output: Journal article publicationJournal articleAcademic researchpeer-review

21 Citations (Scopus)


Predicting unsafe behaviour in advance can enable remedial measures to be put in place to mitigate likely accidents on construction sites. Prevailing safety studies in construction tend to be retrospective and focus on examining the conditions that contribute to unsafe behaviour from a psychological perspective. While such studies are warranted, they can also not visually comprehend the dynamic and complex conditions that influence unsafe behaviour. In this paper, we aim to contribute to filling this void and, in doing so, combine computer vision with Long-Short Term Memory (LSTM) to predict unsafe behaviours from videos automatically. Our proposed approach for predicting unsafe behaviour is based on: (1) tracking people using a SiamMask; (2) predicting the trajectory of people using an improved Social-LSTM; and (3) predicting unsafe behaviour using Franklin's point inclusion polygon (PNPoly) algorithm. We use the Wuhan metro project as a case to evaluate our approach's feasibility and effectiveness. Our adopted SiamMask method outperforms current techniques used for tracking people. Additionally, our improved Social-LSTM can achieve higher accuracy on trajectory prediction than other methods (e.g., Social-GAN). The research findings demonstrate that our developed computer vision approach can be used to accurately predict unsafe behaviour on construction sites.

Original languageEnglish
Article number101400
JournalAdvanced Engineering Informatics
Publication statusPublished - Oct 2021


  • Computer vision
  • Deep learning
  • Long-short term memory
  • PNpoly algorithm
  • Unsafe behaviour

ASJC Scopus subject areas

  • Information Systems
  • Artificial Intelligence


Dive into the research topics of 'Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction'. Together they form a unique fingerprint.

Cite this