Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation

Zhixuan Liang, Jiannong Cao, Shan Jiang, Divya Saxena, Huafeng Xu

Research output: Chapter in book / Conference proceedingConference article published in proceeding or bookAcademic researchpeer-review

11 Citations (Scopus)

Abstract

Many real-world applications can be formulated as multi-agent cooperation problems, such as network packet routing and coordination of autonomous vehicles. The emergence of deep reinforcement learning (DRL) provides a promising approach for multi-agent cooperation through the interaction of the agents and environments. However, traditional DRL solutions suffer from the high dimensions of multiple agents with continuous action space during policy search. Besides, the dynamicity of agents' policies makes the training non-stationary. To tackle the issues, we propose a hierarchical reinforcement learning approach with high-level decision-making and low-level individual control for efficient policy search. In particular, the cooperation of multiple agents can be learned in high-level discrete action space efficiently. At the same time, the low-level individual control can be reduced to single-agent reinforcement learning. In addition to hierarchical reinforcement learning, we propose an opponent modeling network to model other agents' policies during the learning process. In contrast to end-to-end DRL approaches, our approach reduces the learning complexity by decomposing the overall task into sub-tasks in a hierarchical way. To evaluate the efficiency of our approach, we conduct a real-world case study in the cooperative lane change scenario. Both simulation and real-world experiments show the superiority of our approach in the collision rate and convergence speed.

Original languageEnglish
Title of host publicationProceedings - 2022 IEEE 42nd International Conference on Distributed Computing Systems, ICDCS 2022
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages884-894
Number of pages11
ISBN (Electronic)9781665471770
DOIs
Publication statusPublished - Oct 2022
Event42nd IEEE International Conference on Distributed Computing Systems, ICDCS 2022 - Bologna, Italy
Duration: 10 Jul 202213 Jul 2022

Publication series

NameProceedings - International Conference on Distributed Computing Systems
Volume2022-July

Conference

Conference42nd IEEE International Conference on Distributed Computing Systems, ICDCS 2022
Country/TerritoryItaly
CityBologna
Period10/07/2213/07/22

Keywords

  • Deep Reinforcement Learning
  • Hierarchical Reinforcement Learning
  • Multi-agent Cooperation

ASJC Scopus subject areas

  • Software
  • Hardware and Architecture
  • Computer Networks and Communications

Fingerprint

Dive into the research topics of 'Hierarchical Reinforcement Learning with Opponent Modeling for Distributed Multi-agent Cooperation'. Together they form a unique fingerprint.

Cite this