Deep Learning Image Captioning in Construction Management: A Feasibility Study

Bo Xiao, Yiheng Wang, Shih Chung Kang

Research output: Journal article publicationJournal articleAcademic researchpeer-review

1 Citation (Scopus)


Deep learning image captioning methods are able to generate one or several natural sentences to describe the contents of construction images. By deconstructing these sentences, the construction object and activity information can be retrieved integrally for automated scene analysis. However, the feasibility of deep learning image captioning in construction remains unclear. To fill this gap, this research investigates the feasibility of deep learning image captioning methods in construction management. First, a linguistic schema for annotating construction machine images was established, and a captioning data set was developed. Then, six deep learning image captioning methods from the computer vision community were selected and tested on the construction captioning data set. In the sentence-level evaluation, the transformer-self-critical sequence training (Tsfm-SCST) method has obtained the best performance among six methods with the bilingual evaluation (BLEU)-1 score of 0.606, BLEU-2 of 0.506, BLEU-3 of 0.427, BLEU-4 of 0.349, metric for evaluation of translation with explicit ordering (METEOR) of 0.287, recall-oriented understudy for gisting evaluation (ROUGE) of 0.585, consensus-based image description evaluation (CIDEr) of 1.715, and semantic propositional image caption evaluation (SPICE) score of 0.422. In the element-level evaluation, the Tsfm-SCST method achieved an average precision of 91.1%, recall of 83.3%, and an F1 score of 86.6% for recognition of construction machine objects by deconstructing the generated sentences. This research indicates that deep learning image captioning is feasible as a method of generating accurate and precise text descriptions from construction images, with potential applications in construction scene analysis and image documentation.

Original languageEnglish
Article number04022049
JournalJournal of Construction Engineering and Management
Issue number7
Publication statusPublished - 1 Jul 2022


  • Construction machines
  • Deep learning
  • Feasibility study
  • Image captioning
  • Vision-based monitoring

ASJC Scopus subject areas

  • Civil and Structural Engineering
  • Building and Construction
  • Industrial relations
  • Strategy and Management


Dive into the research topics of 'Deep Learning Image Captioning in Construction Management: A Feasibility Study'. Together they form a unique fingerprint.

Cite this