Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks

Bo Peng; Emmanuele Chersoni; Yu-yin Hsu; Chu-ren Huang

doi:10.18653/v1/2021.econlp-1.5

Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks

Department of Language Science and Technology

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

32 Citations (Scopus)

Abstract

With the recent rise in popularity of Transformer models in Natural Language Processing, research efforts have been dedicated to the development of domain-adapted versions of BERT-like architectures. In this study, we focus on FinBERT, a Transformer model trained on text from the financial domain. By comparing its performances with the original BERT on a wide variety of financial text processing tasks, we found continual pretraining from the original model to be the more beneficial option. Domain-specific pretraining from scratch, conversely, seems to be less effective.

Original language	English
Title of host publication	Proceedings of the Third Workshop on Economics and Natural Language Processing
Editors	Udo Hahn, Veronique Hoste, Amanda Stent
Publisher	Association for Computational Linguistics, ACL Anthology
Pages	37–44
Number of pages	8
ISBN (Electronic)	978-1-954085-84-8
DOIs	https://doi.org/10.18653/v1/2021.econlp-1.5
Publication status	Published - Nov 2021
Event	Workshop on Economics and Natural Language Processing - Barcelo Bavaro Convention Center, Punta Cana, Dominican Republic Duration: 11 Nov 2021 → 11 Nov 2021

Workshop

Workshop	Workshop on Economics and Natural Language Processing
Country/Territory	Dominican Republic
City	Punta Cana
Period	11/11/21 → 11/11/21

More information

10.18653/v1/2021.econlp-1.5

Cite this

Peng, B., Chersoni, E., Hsu, Y., & Huang, C. (2021). Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks. In U. Hahn, V. Hoste, & A. Stent (Eds.), Proceedings of the Third Workshop on Economics and Natural Language Processing (pp. 37–44). Association for Computational Linguistics, ACL Anthology. https://doi.org/10.18653/v1/2021.econlp-1.5

@inproceedings{6dbedc45d8a64a5b9aa7bfe69215a0e4,

title = "Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks",

abstract = "With the recent rise in popularity of Transformer models in Natural Language Processing, research efforts have been dedicated to the development of domain-adapted versions of BERT-like architectures. In this study, we focus on FinBERT, a Transformer model trained on text from the financial domain. By comparing its performances with the original BERT on a wide variety of financial text processing tasks, we found continual pretraining from the original model to be the more beneficial option. Domain-specific pretraining from scratch, conversely, seems to be less effective.",

author = "Bo Peng and Emmanuele Chersoni and Yu-yin Hsu and Chu-ren Huang",

year = "2021",

month = nov,

doi = "10.18653/v1/2021.econlp-1.5",

language = "English",

pages = "37–44",

editor = "Udo Hahn and Veronique Hoste and Amanda Stent",

booktitle = "Proceedings of the Third Workshop on Economics and Natural Language Processing",

publisher = "Association for Computational Linguistics, ACL Anthology",

note = "Workshop on Economics and Natural Language Processing ; Conference date: 11-11-2021 Through 11-11-2021",

}

Peng, B, Chersoni, E , Hsu, Y & Huang, C 2021, Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks. in U Hahn, V Hoste & A Stent (eds), Proceedings of the Third Workshop on Economics and Natural Language Processing. Association for Computational Linguistics, ACL Anthology, pp. 37–44, Workshop on Economics and Natural Language Processing, Punta Cana, Dominican Republic, 11/11/21. https://doi.org/10.18653/v1/2021.econlp-1.5

Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks. / Peng, Bo; Chersoni, Emmanuele ; Hsu, Yu-yin et al.
Proceedings of the Third Workshop on Economics and Natural Language Processing. ed. / Udo Hahn; Veronique Hoste; Amanda Stent. Association for Computational Linguistics, ACL Anthology, 2021. p. 37–44.

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks

AU - Peng, Bo

AU - Chersoni, Emmanuele

AU - Hsu, Yu-yin

AU - Huang, Chu-ren

PY - 2021/11

Y1 - 2021/11

N2 - With the recent rise in popularity of Transformer models in Natural Language Processing, research efforts have been dedicated to the development of domain-adapted versions of BERT-like architectures. In this study, we focus on FinBERT, a Transformer model trained on text from the financial domain. By comparing its performances with the original BERT on a wide variety of financial text processing tasks, we found continual pretraining from the original model to be the more beneficial option. Domain-specific pretraining from scratch, conversely, seems to be less effective.

AB - With the recent rise in popularity of Transformer models in Natural Language Processing, research efforts have been dedicated to the development of domain-adapted versions of BERT-like architectures. In this study, we focus on FinBERT, a Transformer model trained on text from the financial domain. By comparing its performances with the original BERT on a wide variety of financial text processing tasks, we found continual pretraining from the original model to be the more beneficial option. Domain-specific pretraining from scratch, conversely, seems to be less effective.

U2 - 10.18653/v1/2021.econlp-1.5

DO - 10.18653/v1/2021.econlp-1.5

M3 - Conference article published in proceeding or book

SP - 37

EP - 44

BT - Proceedings of the Third Workshop on Economics and Natural Language Processing

A2 - Hahn, Udo

A2 - Hoste, Veronique

A2 - Stent, Amanda

PB - Association for Computational Linguistics, ACL Anthology

T2 - Workshop on Economics and Natural Language Processing

Y2 - 11 November 2021 through 11 November 2021

ER -

Is Domain Adaptation Worth Your Investment? Comparing BERT and FinBERT on Financial Tasks

Abstract

Workshop

More information

Fingerprint

Cite this