To Cooperate or Not: Multi-Agent Reinforcement Learning-Based On-Ramp Merging Strategies for Autonomous Vehicles

On-ramp merging for Autonomous Vehicles (AVs) is a challenging task that involves complex interactions of multiple vehicles and should consider both the safety and efficiency. Extensive works have been conducted on merging decisions using various control and learning-based methods. Although these methods are proved effective in different scenarios, debate still exist whether AVs should cooperate or not during the processes, or whether such cooperation should be considered in the method design. To answer these questions, this paper formulates the problem of on-ramp merging for AVs as a Partially Observable Markov Decision Process (POMDP), and develops several cooperative and non-cooperative Multi-Agent Reinforcement Learning (MARL) algorithms based on state-of-the-art models, i.e., Independent DQN, Bi-level Actor-Critic, and MADDPG. Experimental case studies are conducted to compare the performance of these algorithms, in terms of training efficiency, merge rate, average speed, and safety metrics (such as collision rates and acceleration). Surprisingly, results reveal that it is not always good to cooperate in merging process and the non-cooperation method performs better in many cases. The findings also emphasize the importance of strategy selection based on specific traffic scenarios and objectives. Besides, by understanding the interplay between cooperation and independent optimization, our work provides insights for safer and more efficient traffic management.