TY - GEN
T1 - Effectiveness of Pre-training for Few-shot Intent Classification
AU - Zhang, Haode
AU - Zhang, Yuwei
AU - Zhan, Li Ming
AU - Chen, Jiaxin
AU - Shi, Guangyuan
AU - Wu, Xiao Ming
AU - Lam, Albert Y.S.
N1 - Funding Information:
This research was supported by the grants of HK ITF UIM/377 and DaSAIL project P0030935.
Publisher Copyright:
© 2021 Association for Computational Linguistics.
PY - 2021/9
Y1 - 2021/9
N2 - This paper investigates the effectiveness of pre-training for few-shot intent classification. While existing paradigms commonly further pre-train language models such as BERT on a vast amount of unlabeled corpus, we find it highly effective and efficient to simply finetune BERT with a small set of labeled utterances from public datasets. Specifically, fine-tuning BERT with roughly 1,000 labeled data yields a pre-trained model - IntentBERT, which can easily surpass the performance of existing pre-trained models for few-shot intent classification on novel domains with very different semantics. The high effectiveness of IntentBERT confirms the feasibility and practicality of few-shot intent detection, and its high generalization ability across different domains suggests that intent classification tasks may share a similar underlying structure, which can be efficiently learned from a small set of labeled data. The source code can be found at https://github.com/ hdzhang-code/IntentBERT.
AB - This paper investigates the effectiveness of pre-training for few-shot intent classification. While existing paradigms commonly further pre-train language models such as BERT on a vast amount of unlabeled corpus, we find it highly effective and efficient to simply finetune BERT with a small set of labeled utterances from public datasets. Specifically, fine-tuning BERT with roughly 1,000 labeled data yields a pre-trained model - IntentBERT, which can easily surpass the performance of existing pre-trained models for few-shot intent classification on novel domains with very different semantics. The high effectiveness of IntentBERT confirms the feasibility and practicality of few-shot intent detection, and its high generalization ability across different domains suggests that intent classification tasks may share a similar underlying structure, which can be efficiently learned from a small set of labeled data. The source code can be found at https://github.com/ hdzhang-code/IntentBERT.
UR - http://www.scopus.com/inward/record.url?scp=85129197298&partnerID=8YFLogxK
M3 - Conference article published in proceeding or book
AN - SCOPUS:85129197298
T3 - Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021
SP - 1114
EP - 1120
BT - Findings of the Association for Computational Linguistics, Findings of ACL
A2 - Moens, Marie-Francine
A2 - Huang, Xuanjing
A2 - Specia, Lucia
A2 - Yih, Scott Wen-Tau
PB - Association for Computational Linguistics (ACL)
T2 - 2021 Findings of the Association for Computational Linguistics, Findings of ACL: EMNLP 2021
Y2 - 7 November 2021 through 11 November 2021
ER -