A hybrid penalty method for a class of optimization problems with multiple rank constraints

In this paper, we consider the problem of minimizing a smooth objective over multiple rank constraints on Hankel structured matrices. These kinds of problems arise in system identification, system theory, and signal processing, where the rank constraints are typically "hard constraints.""To solve these problems, we propose a hybrid penalty method that combines a penalty method with a postprocessing scheme. Specifically, we solve the penalty subproblems until the penalty parameter reaches a given threshold, and then switch to a local alternating "pseudoprojection""method to further reduce constraint violation. Pseudoprojection is a generalization of the concept of projection. We show that a pseudoprojection onto a single low-rank Hankel structured matrix constraint can be computed efficiently by existing software such as SLRA [I. Markovsky and K. Usevich, J. Comput. Appl. Math., 256 (2014), pp. 278-292], under mild assumptions. We also demonstrate how the penalty subproblems in the hybrid penalty method can be solved by pseudoprojection-based optimization methods, and then present some convergence results for our hybrid penalty method. Finally, the efficiency of our method is illustrated by numerical examples.