Cognitive Support in Aviation Operations Utilizing Multimodal Large Language Models

Cognitive support in aviation operations can help reduce cognitive workload and improve flight safety. Existing data-driven models, typically trained for specific aircraft types, often lack generalization capabilities and cannot be directly applied to other aircrafts. The multimodal large language models (MLLMs) have shown significant value in various applications without additional training. Motivated by the success of MLLMs, we explore and develop a proactive aviation cognitive support framework (PACS) based on MLLMs to proactively assist pilots in anomaly perception and decision-making during aviation emergencies. We propose a localization-augmented anomaly perception method to address the limitations of MLLMs in recognizing small-sized warning message text. To improve the accuracy and efficiency in decision-making, we design a hierarchically structured aviation knowledge base and a "Retrieval-Selection Generation"strategy to generate accurate operational instructions. The experimental results demonstrate high accuracy in anomaly perception (87.51%) and decision-making (93.04%), while reducing token count by up to 56.77%. Our PACS is expected to adapt to various scenarios without additional fine-tuning, offering a new paradigm for future human-AI collaboration in aviation operations.