TY - GEN
T1 - A closer look at the training strategy for modern meta-learning
AU - Chen, Jiaxin
AU - Wu, Xiao Ming
AU - Li, Yanke
AU - Li, Qimai
AU - Zhan, Li Ming
AU - Chung, Fu Lai
N1 - Funding Information:
The authors would like to thank Junjie Ye for helpful discussion and the anonymous reviewers for their valuable comments. This research was supported by the grants of DaSAIL projects P0030935 and P0030970 funded by PolyU (UGC).
Publisher Copyright:
© 2020 Neural information processing systems foundation. All rights reserved.
PY - 2020/12
Y1 - 2020/12
N2 - The support/query (S/Q) episodic training strategy has been widely used in modern meta-learning algorithms and is believed to improve their generalization ability to test environments. This paper conducts a theoretical investigation of this training strategy on generalization. From a stability perspective, we analyze the generalization error bound of generic meta-learning algorithms trained with such strategy. We show that the S/Q episodic training strategy naturally leads to a counterintuitive generalization bound of O(1/√n), which only depends on the task number n but independent of the inner-task sample size m. Under the common assumption m << n for few-shot learning, the bound of O(1/√n) implies strong generalization guarantees for modern meta-learning algorithms in the few-shot regime. To further explore the influence of training strategies on generalization, we propose a leave-one-out (LOO) training strategy for meta-learning and compare it with S/Q training. Experiments on standard few-shot regression and classification tasks with popular meta-learning algorithms validate our analysis.
AB - The support/query (S/Q) episodic training strategy has been widely used in modern meta-learning algorithms and is believed to improve their generalization ability to test environments. This paper conducts a theoretical investigation of this training strategy on generalization. From a stability perspective, we analyze the generalization error bound of generic meta-learning algorithms trained with such strategy. We show that the S/Q episodic training strategy naturally leads to a counterintuitive generalization bound of O(1/√n), which only depends on the task number n but independent of the inner-task sample size m. Under the common assumption m << n for few-shot learning, the bound of O(1/√n) implies strong generalization guarantees for modern meta-learning algorithms in the few-shot regime. To further explore the influence of training strategies on generalization, we propose a leave-one-out (LOO) training strategy for meta-learning and compare it with S/Q training. Experiments on standard few-shot regression and classification tasks with popular meta-learning algorithms validate our analysis.
UR - http://www.scopus.com/inward/record.url?scp=85102047940&partnerID=8YFLogxK
M3 - Conference article published in proceeding or book
AN - SCOPUS:85102047940
VL - 2020-December
T3 - Advances in Neural Information Processing Systems
SP - 1
EP - 11
BT - 34th Conference on Neural Information Processing Systems, NeurIPS 2020
T2 - 34th Conference on Neural Information Processing Systems, NeurIPS 2020
Y2 - 6 December 2020 through 12 December 2020
ER -