Multi-Timescale Reward-Based DRL Energy Management for Regenerative Braking Energy Storage System

The traditional model-based energy management strategy (EMS) for regenerative braking energy storage systems (RBESSs) is obsoleting in the face of increasingly complex and uncertain operation conditions in railway power systems (RPSs). In this article, a model-free deep reinforcement learning (DRL) method is proposed. First, the multiobjective energy management problem for RBESS is formulated to concurrently achieve the regenerative braking energy (RBE) utilization and power demand shaving of RPS. Then, this problem is modeled as a Markov decision process (MDP) to be solved by the DRL-based method. Specifically, the RBESS controller is modeled as an agent to interact with the environment modeled as the RPS integrated with RBESS. To coordinate the agent to learn the optimal strategies regarding multiple energy management objectives in different timescales, a multistage reward function (MSRF) involving the step reward and final reward is designed. Based on the above elements, the double deep Q-learning algorithm is applied to train the agent for optimizing the EMS. Finally, the proposed DRL-based EMS is tested on the OPAL-RT experimental platform by using the field load data. Case studies have demonstrated that the proposed method outperforms the traditional rule-based and optimization-based methods by over 5% in the energy management objective.