Data-driven optimal control of nonlinear systems via Carleman operator and value iteration

For solving the optimal control problem of nonlinear systems, conventional adaptive dynamic programming (ADP) methods always exploit neural networks to approximate the nonlinear functions in the Hamilton-Jacobi-Bellman (HJB) equation. However, the selection of basis functions in neural networks remains a challenging task, especially when the model dynamics are unknown. This article develops a novel data-driven ADP scheme for the optimal control problem of nonlinear polynomial systems with totally unknown dynamics. We utilize the Carleman operator to lift the nonlinear model into an infinite-dimensional bilinear system and the original optimal control problem is transformed into solving the infinite-dimensional Riccati equation. Then, by using state and input measurements, we present a value iteration-based algorithm to learn the near-optimal control protocol which comes from a finite-dimensional truncated Riccati equation. Besides, we prove that, under routine conditions, the obtained controller stabilizes the closed-loop system and its robustness property is analyzed. It should be noted that the proposed methodology significantly circumvents the issue of specifying basis functions in the existing ADP-based optimal control schemes and gives rise to computational improvement. Finally, three application examples are provided to demonstrate the feasibility of our method.