Operational optimization for the grid-connected residential photovoltaic-battery system using model-based reinforcement learning

The development of distributed photovoltaic and energy storage devices has created challenges for energy management systems due to uncertainty and mismatch between local generation and residents' energy demand. Reinforcement learning is gaining attention as a control algorithm, but traditional model-free RL has data quality and quantity limitations for energy management applications. Therefore, this study proposed a model-based deep RL method to optimize the operation control of the energy storage system by taking the measured dataset of an actual existing building in Japan as the research object. With an optimization goal of reducing the microgrid's energy cost and ensuring the PV self-consumption ratio, we designed a new reward function for these goals. We took the benchmark strategy currently used by the target building's energy management system as the baseline model in the experiment. We applied four advanced RL algorithms (PPO, DQN, DDPG, and TD3) to optimize the baseline model. The results show that the proposed RL design can better achieve the two optimization objectives of minimizing energy cost and maximizing the PV self-consumption ratio. Among them, the TD3 algorithm presented the best performance. Compared with the baseline model, its annual energy cost can be reduced by 17.82%, and the photovoltaic self-consumption ratio can be increased by 0.86%. In addition, the model-based RL method proposed in this paper can provide a better energy management strategy with the training set of only one and a half years of measured data, which proves that it has a high potential for practical application.