TY - GEN
T1 - Multi-objective Neural Architecture Search with Almost No Training
AU - Hu, Shengran
AU - Cheng, Ran
AU - He, Cheng
AU - Lu, Zhichao
N1 - Publisher Copyright:
© 2021, Springer Nature Switzerland AG.
PY - 2021
Y1 - 2021
N2 - In the recent past, neural architecture search (NAS) has attracted increasing attention from both academia and industries. Despite the steady stream of impressive empirical results, most existing NAS algorithms are computationally prohibitive to execute due to the costly iterations of stochastic gradient descent (SGD) training. In this work, we propose an effective alternative, dubbed Random-Weight Evaluation (RWE), to rapidly estimate the performance of network architectures. By just training the last linear classification layer, RWE reduces the computational cost of evaluating an architecture from hours to seconds. When integrated within an evolutionary multi-objective algorithm, RWE obtains a set of efficient architectures with state-of-the-art performance on CIFAR-10 with less than two hours’ searching on a single GPU card. Ablation studies on rank-order correlations and transfer learning experiments to ImageNet have further validated the effectiveness of RWE.
AB - In the recent past, neural architecture search (NAS) has attracted increasing attention from both academia and industries. Despite the steady stream of impressive empirical results, most existing NAS algorithms are computationally prohibitive to execute due to the costly iterations of stochastic gradient descent (SGD) training. In this work, we propose an effective alternative, dubbed Random-Weight Evaluation (RWE), to rapidly estimate the performance of network architectures. By just training the last linear classification layer, RWE reduces the computational cost of evaluating an architecture from hours to seconds. When integrated within an evolutionary multi-objective algorithm, RWE obtains a set of efficient architectures with state-of-the-art performance on CIFAR-10 with less than two hours’ searching on a single GPU card. Ablation studies on rank-order correlations and transfer learning experiments to ImageNet have further validated the effectiveness of RWE.
KW - Evolutionary algorithms
KW - Multi-objective optimization
KW - Neural architecture search
KW - Performance estimation
UR - https://www.scopus.com/pages/publications/85107284976
U2 - 10.1007/978-3-030-72062-9_39
DO - 10.1007/978-3-030-72062-9_39
M3 - Conference article published in proceeding or book
AN - SCOPUS:85107284976
SN - 9783030720612
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 492
EP - 503
BT - Evolutionary Multi-Criterion Optimization - 11th International Conference, EMO 2021, Proceedings
A2 - Ishibuchi, Hisao
A2 - Zhang, Qingfu
A2 - Cheng, Ran
A2 - Li, Ke
A2 - Li, Hui
A2 - Wang, Handing
A2 - Zhou, Aimin
PB - Springer Science and Business Media Deutschland GmbH
T2 - 11th International Conference on Evolutionary Multi-Criterion Optimization, EMO 2021
Y2 - 28 March 2021 through 31 March 2021
ER -