Bilevel Optimization with Convex Majorant Approach for Training Sparse Neural Networks

This paper proposes a convex majorant approach for training sparse neural networks by bilevel optimization, where the upper level problem minimizes a smooth nonconvex function, and the lower level problem minimizes a smooth nonconvex function with a nonsmooth convex group sparse regularizer over a box set for fixed sparse regularization hyperparameters. The convex majorant function approximates the objective function of the lower level problem. We establish the relationship between the original bilevel optimization and the bilevel optimization with the convex majorant approach regarding global and local minimizers. Moreover, we use a smoothing function to approximate the convex majorant function and derive the convergence of global minimizers to those of the corresponding nonsmooth bilevel problems with smoothing parameter converging to zero. A smoothing implicit function method is proposed to solve the smooth approximate bilevel optimization problem. Some numerical experiments including the tests on the data from machine learning repository show that the convex majorant approach performs better than the widely used Grid Search method, Random Search method, and Bayesian optimization method.