Enhancing Readback Verification in ATCOs-Pilot Communication Using Audio-Visual Interaction: Insights from fNIRS and Eye-Tracking

Miscommunications between air traffic controllers (ATCOs) and pilots can lead to severe accidents, making accurate readback verification essential. Audio-visual interaction, namely generative subtitles, is expected to enhance readback verification by providing redundant information. However, this redundancy may also increase ATCOs’ cognitive workload and challenge their working memory capacity. This study investigated the effect of subtitles on readback performance and cognitive workload across different working memory capacities. A simulated verification task based on ATCO task analysis was employed, and functional near-infrared spectroscopy with eye-tracking was analyzed to assess subtitles’ effects. Results showed that subtitles improved readback performance across working memory groups without increasing cognitive workload. Oxygenated hemoglobin deactivation in frontal, parietal, and occipital regions indicated more efficient cognitive processing, and higher blink rates among high working memory participants suggested reduced cognitive load. This study deepens understanding of audio-visual interaction and offers practical insights for enhancing ATCO–pilot communication.