Recurrent neural network with pooling operation and attention mechanism for sentiment analysis: A multi-task learning approach

Yi Cai, Qingbao Huang, Zejun Lin, Jingyun Xu, Zhenhong Chen, Qing Li

Research output: Journal article publicationJournal articleAcademic researchpeer-review

37 Citations (Scopus)


Sentiment analysis is designed to classify documents into a fixed number of pre-defined categories that represent different sentiments. Focusing on the limitation of insufficient training data, multi-task learning models based on deep learning have recently achieved significant progress in this field. In general, these models leverage multiple datasets annotated for different tasks to improve the performance on each individual dataset. The improvement is particularly evident on tasks with limited training data. However, most of these models suffer from two limitations. First, they use the final output of the hidden layer as the overall representation of the text, which initially loses a certain amount of semantic information. Second, although some of them utilize a certain gate mechanism to select shared features, some irrelevant shared features are erroneously used owing to polysemy. To address these two limitations, we integrate a pooling layer into a Bi-directional Recurrent Neural Network (BRNN) to extract semantic information comprehensively. We then apply the attention mechanism between shared layers and task-specific layers to identify the effective shared features, and propose an Attention-based Separate Pooling BRNN (ASP-BRNN) model. We conduct experiments to show the effectiveness of our models on four datasets (SST1, SST2, SUBJ, and IMDB), and the accuracy of our models increases steadily by approximately 0.5% for each model. It proves the effectiveness of every newly added component in solving the two problems. A further evaluation on eight datasets shows our proposed ASP-BRNN model outperforms current state-of-the-art models, such as ASP-MTL model (at least +0.2% on Electronics and at most +6.9% on IMDB), MT-ARC-II model (at least +0.2% on SST2 and at most +3.8% on DVDs).

Original languageEnglish
Article number105856
Pages (from-to)1-12
Number of pages12
JournalKnowledge-Based Systems
Publication statusPublished - 5 Sept 2020


  • Attention mechanism
  • Multi-task
  • Opinion mining
  • Recurrent neural network
  • Sentiment analysis

ASJC Scopus subject areas

  • Management Information Systems
  • Software
  • Information Systems and Management
  • Artificial Intelligence


Dive into the research topics of 'Recurrent neural network with pooling operation and attention mechanism for sentiment analysis: A multi-task learning approach'. Together they form a unique fingerprint.

Cite this