TY - JOUR
T1 - Evolving Deep Neural Networks via Cooperative Coevolution with Backpropagation
AU - Gong, Maoguo
AU - Liu, Jia
AU - Qin, A. K.
AU - Zhao, Kun
AU - Tan, Kay Chen
N1 - Funding Information:
Manuscript received June 20, 2019; revised December 3, 2019; accepted March 2, 2020. Date of publication March 25, 2020; date of current version January 5, 2021. This work was supported in part by the National Nature Science Foundation of China under Grant 61772393, in part by the Key Research and Development Program of Shaanxi Province under Grant 2018ZDXM-GY-045, in part by the Australian Research Council under Grant LP170100416, Grant LP180100114, and Grant DP200102611, and in part by the Research Grants Council of the Hong Kong Special Administrative Region, China, under Grant CityU11202418 and Grant CityU11209219. (Corresponding author: Maoguo Gong.) Maoguo Gong and Kun Zhao are with the Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, Xidian University, Xi’an 710071, China (e-mail: [email protected]).
Publisher Copyright:
© 2012 IEEE.
PY - 2021/1
Y1 - 2021/1
N2 - Deep neural networks (DNNs), characterized by sophisticated architectures capable of learning a hierarchy of feature representations, have achieved remarkable successes in various applications. Learning DNN's parameters is a crucial but challenging task that is commonly resolved by using gradient-based backpropagation (BP) methods. However, BP-based methods suffer from severe initialization sensitivity and proneness to getting trapped into inferior local optima. To address these issues, we propose a DNN learning framework that hybridizes CC-based optimization with BP-based gradient descent, called BPCC, and implement it by devising a computationally efficient CC-based optimization technique dedicated to DNN parameter learning. In BPCC, BP will intermittently execute for multiple training epochs. Whenever the execution of BP in a training epoch cannot sufficiently decrease the training objective function value, CC will kick in to execute by using the parameter values derived by BP as the starting point. The best parameter values obtained by CC will act as the starting point of BP in its next training epoch. In CC-based optimization, the overall parameter learning task is decomposed into many subtasks of learning a small portion of parameters. These subtasks are individually addressed in a cooperative manner. In this article, we treat neurons as basic decomposition units. Furthermore, to reduce the computational cost, we devise a maturity-based subtask selection strategy to selectively solve some subtasks of higher priority. Experimental results demonstrate the superiority of the proposed method over common-practice DNN parameter learning techniques.
AB - Deep neural networks (DNNs), characterized by sophisticated architectures capable of learning a hierarchy of feature representations, have achieved remarkable successes in various applications. Learning DNN's parameters is a crucial but challenging task that is commonly resolved by using gradient-based backpropagation (BP) methods. However, BP-based methods suffer from severe initialization sensitivity and proneness to getting trapped into inferior local optima. To address these issues, we propose a DNN learning framework that hybridizes CC-based optimization with BP-based gradient descent, called BPCC, and implement it by devising a computationally efficient CC-based optimization technique dedicated to DNN parameter learning. In BPCC, BP will intermittently execute for multiple training epochs. Whenever the execution of BP in a training epoch cannot sufficiently decrease the training objective function value, CC will kick in to execute by using the parameter values derived by BP as the starting point. The best parameter values obtained by CC will act as the starting point of BP in its next training epoch. In CC-based optimization, the overall parameter learning task is decomposed into many subtasks of learning a small portion of parameters. These subtasks are individually addressed in a cooperative manner. In this article, we treat neurons as basic decomposition units. Furthermore, to reduce the computational cost, we devise a maturity-based subtask selection strategy to selectively solve some subtasks of higher priority. Experimental results demonstrate the superiority of the proposed method over common-practice DNN parameter learning techniques.
KW - Backpropagation (BP)
KW - cooperative coevolution (CC)
KW - deep neural networks (DNNs)
KW - evolutionary optimization
UR - http://www.scopus.com/inward/record.url?scp=85099132272&partnerID=8YFLogxK
U2 - 10.1109/TNNLS.2020.2978857
DO - 10.1109/TNNLS.2020.2978857
M3 - Journal article
C2 - 32217489
AN - SCOPUS:85099132272
SN - 2162-237X
VL - 32
SP - 420
EP - 434
JO - IEEE Transactions on Neural Networks and Learning Systems
JF - IEEE Transactions on Neural Networks and Learning Systems
IS - 1
ER -