Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

Xin Wu; Yi Cai; Qing Li; Jingyun Xu; Ho fung Leung

doi:10.1007/978-3-030-02922-7_31

Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

Xin Wu, Yi Cai, Qing Li, Jingyun Xu, Ho fung Leung

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

16 Citations (Scopus)

Abstract

Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.

Original language	English
Title of host publication	Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings
Editors	Hye-Young Paik, Hua Wang, Rui Zhou, Hakim Hacid, Wojciech Cellary
Publisher	Springer-Verlag
Pages	453-467
Number of pages	15
ISBN (Print)	9783030029210
DOIs	https://doi.org/10.1007/978-3-030-02922-7_31
Publication status	Published - 1 Jan 2018
Externally published	Yes
Event	19th International Conference on Web Information Systems Engineering, WISE 2018 - Dubai, United Arab Emirates Duration: 12 Nov 2018 → 15 Nov 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11233 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	19th International Conference on Web Information Systems Engineering, WISE 2018
Country/Territory	United Arab Emirates
City	Dubai
Period	12/11/18 → 15/11/18

Keywords

Attention mechanism
Convolutional neural networks
Text classification
Word representation

ASJC Scopus subject areas

Theoretical Computer Science
General Computer Science

More information

10.1007/978-3-030-02922-7_31

Cite this

Wu, X., Cai, Y., Li, Q., Xu, J., & Leung, H. F. (2018). Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. In H.-Y. Paik, H. Wang, R. Zhou, H. Hacid, & W. Cellary (Eds.), Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings (pp. 453-467). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11233 LNCS). Springer-Verlag. https://doi.org/10.1007/978-3-030-02922-7_31

Wu, Xin ; Cai, Yi ; Li, Qing et al. / Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings. editor / Hye-Young Paik ; Hua Wang ; Rui Zhou ; Hakim Hacid ; Wojciech Cellary. Springer-Verlag, 2018. pp. 453-467 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{920106b60caa4afbb9bfe4edad922b85,

title = "Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification",

abstract = "Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.",

keywords = "Attention mechanism, Convolutional neural networks, Text classification, Word representation",

author = "Xin Wu and Yi Cai and Qing Li and Jingyun Xu and Leung, {Ho fung}",

year = "2018",

month = jan,

day = "1",

doi = "10.1007/978-3-030-02922-7_31",

language = "English",

isbn = "9783030029210",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer-Verlag",

pages = "453--467",

editor = "Hye-Young Paik and Hua Wang and Rui Zhou and Hakim Hacid and Wojciech Cellary",

booktitle = "Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings",

note = "19th International Conference on Web Information Systems Engineering, WISE 2018 ; Conference date: 12-11-2018 Through 15-11-2018",

}

Wu, X, Cai, Y, Li, Q, Xu, J & Leung, HF 2018, Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. in H-Y Paik, H Wang, R Zhou, H Hacid & W Cellary (eds), Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11233 LNCS, Springer-Verlag, pp. 453-467, 19th International Conference on Web Information Systems Engineering, WISE 2018, Dubai, United Arab Emirates, 12/11/18. https://doi.org/10.1007/978-3-030-02922-7_31

Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. / Wu, Xin; Cai, Yi; Li, Qing et al.
Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings. ed. / Hye-Young Paik; Hua Wang; Rui Zhou; Hakim Hacid; Wojciech Cellary. Springer-Verlag, 2018. p. 453-467 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11233 LNCS).

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

AU - Wu, Xin

AU - Cai, Yi

AU - Li, Qing

AU - Xu, Jingyun

AU - Leung, Ho fung

PY - 2018/1/1

Y1 - 2018/1/1

N2 - Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.

AB - Convolutional neural networks (CNN) are widely used in many NLP tasks, which can employ convolutional filters to capture useful semantic features of texts. However, convolutional filters with small window size may lose global context information of texts, simply increasing window size will bring the problems of data sparsity and enormous parameters. To capture global context information, we propose to use the self-attention mechanism to obtain contextual word embeddings. We present two methods to combine word and contextual embeddings, then apply convolutional neural networks to capture semantic features. Experimental results on five commonly used datasets show the effectiveness of our proposed methods.

KW - Attention mechanism

KW - Convolutional neural networks

KW - Text classification

KW - Word representation

UR - http://www.scopus.com/inward/record.url?scp=85055802931&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-02922-7_31

DO - 10.1007/978-3-030-02922-7_31

M3 - Conference article published in proceeding or book

AN - SCOPUS:85055802931

SN - 9783030029210

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 453

EP - 467

BT - Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings

A2 - Paik, Hye-Young

A2 - Wang, Hua

A2 - Zhou, Rui

A2 - Hacid, Hakim

A2 - Cellary, Wojciech

PB - Springer-Verlag

T2 - 19th International Conference on Web Information Systems Engineering, WISE 2018

Y2 - 12 November 2018 through 15 November 2018

ER -

Wu X, Cai Y, Li Q, Xu J, Leung HF. Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification. In Paik HY, Wang H, Zhou R, Hacid H, Cellary W, editors, Web Information Systems Engineering – WISE 2018 - 19th International Conference, 2018, Proceedings. Springer-Verlag. 2018. p. 453-467. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-02922-7_31

Combining Contextual Information by Self-attention Mechanism in Convolutional Neural Networks for Text Classification

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

Cite this