Skip to main navigation Skip to search Skip to main content

BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

Abstract

Pretrained transformer-based models, such as BERT and its variants, have become a common choice to obtain state-of-the-art performances in NLP tasks. In the identification of Adverse Drug Events (ADE) from social media texts, for example, BERT architectures rank first in the leaderboard. However, a systematic comparison between these models has not yet been done. In this paper, we aim at shedding light on the differences between their performance analyzing the results of 12 models, tested on two standard benchmarks. SpanBERT and PubMedBERT emerged as the best models in our evaluation: this result clearly shows that span-based pretraining gives a decisive advantage in the precise recognition of ADEs, and that in-domain language pretraining is particularly useful when the transformer model is trained just on biomedical text from scratch.
Original languageEnglish
Title of host publicationProceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics
EditorsPaola Merlo, Jorg Tiedemann, Reut Tsarfaty
PublisherAssociation for Computational Linguistics (ACL)
Pages1740-1747
ISBN (Electronic)9781954085022
DOIs
Publication statusPublished - Apr 2021
EventThe 16th Conference of the European Chapter of the Association for Computational Linguistics - Online
Duration: 19 Apr 2021 → 23 Apr 2021
https://2021.eacl.org/

Conference

ConferenceThe 16th Conference of the European Chapter of the Association for Computational Linguistics
Period19/04/21 → 23/04/21
Internet address

Fingerprint

Dive into the research topics of 'BERT Prescriptions to Avoid Unwanted Headaches: A Comparison of Transformer Architectures for Adverse Drug Event Detection'. Together they form a unique fingerprint.

Cite this