Game-Theoretic Reinforcement Learning-Based Behavior-Aware Merging in Mixed Traffic

With the increasing integration of Autonomous Vehicles (AVs) into traffic systems, the interaction between different types of vehicles presents a significant challenge for cooperative decision-making, particularly in mixed traffic environments where AVs coexist with Human-driven Vehicles (HVs). Among various scenarios, highway on-ramp merging is a critical and complex one where effective interaction is essential for ensuring traffic safety and efficiency, and decisions of AVs and HVs should be made efficiently. In this paper, we propose a Game-Theoretic Reinforcement Learning (GTRL)-based vehicle behavior interaction framework designed for various merging scenarios in mixed traffic environments. This framework includes Stackelberg leader-follower interactions between AVs and HVs, as well as Nash cooperative interactions between vehicles of the same type, such as AV-AV or HV-HV. We formulate the problem as a Partially Observable Markov Decision Process (POMDP) and employ the Bi-level Reinforcement Learning (Bi-RL) method and Safe Multi-Agent Deep Q-Network (MADQN) method to address two distinct types of interaction problems. A gym-based simulation environment is developed to evaluate four interaction scenarios between vehicles in mixed traffic. Comprehensive experimental results demonstrate the potential of the GTRL-based behavior interaction framework to adapt to the dynamic and uncertain nature of vehicle interactions.