DriveCompanion: Leveraging Vision-Language Models and Retrieval Augmented Generation for Enhancing Driving Convenience

With the increasing complexity of modern road environments, drivers frequently encounter unfamiliar scenarios where understanding and complying with local traffic rules becomes challenging. Traditional navigation systems are limited in their ability to interpret complex real-time scenes or provide context-aware guidance tailored to specific situations, leaving drivers at risk of making errors. The advent of Vision-Language Models (VLMs) offers a promising solution by enabling simultaneous processing and understanding of both visual inputs and user queries. Meanwhile, Retrieval-Augmented Generation (RAG) enhances the generation of driving strategies by retrieving relevant information from external knowledge bases, addressing issues such as hallucinations, outdated knowledge, and lack of transparency inherent in VLMs. To address these challenges, DriveCompanion is proposed as a smart driving assistant that integrates VLMs and RAG. It interprets real-time driving scenes using VLMs while dynamically retrieving scenario-specific traffic rules through RAG, enabling users to inquire about and receive personalized, adaptive driving guidance in unfamiliar situations. A case study in an urban driving scenario demonstrates its effectiveness in improving driving safety and convenience, while an ablation study highlights its superior performance in adaptability and accurate guidance. This system represents a significant step toward smarter, safer and more user-centric driving assistance.