SentiLARE: Sentiment-Aware Language Representation Learning with Linguistic Knowledge
Pei Ke, Haozhe Ji∗, Siyang Liu, Xiaoyan Zhu, Minlie Huang Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China Equal contribution Corresponding author
Abstract
Most of the existing pre-trained language representation models neglect to consider the linguistic knowledge of texts, which can promote language understanding in NLP tasks. To benefit the downstream tasks in sentiment analysis, we propose a novel language representation model called SentiLARE, which introduces word-level linguistic knowledge including part-of-speech tag and sentiment polarity (inferred from SentiWordNet) into pre-trained models. We first propose a context-aware sentiment attention mechanism to acquire the sentiment polarity of each word with its part-of-speech tag by querying SentiWordNet. Then, we devise a new pre-training task called label-aware masked language model to construct knowledge-aware language representation. Experiments show that SentiLARE obtains new state-of-the-art performance on a variety of sentiment analysis tasks111The data, codes, and model parameters are available at https://github.com/thu-coai/SentiLARE..
中文速览
情感分析任务中，现有预训练语言模型（如BERT）在预训练阶段完全忽视了词性、情感极性等语言学知识，导致模型对情感语义的理解能力受限。为此，研究者提出了SentiLARE这一新型预训练模型，先用一种上下文感知的情感注意力机制从SentiWordNet中动态推断每个词的词性标注与情感极性，再设计"标签感知掩码语言模型"（label-aware masked language model）这一预训练任务，通过"早期融合"和"后期监督"两个子任务将句子级情感标签与词级语言学知识深度绑定在一起。在句子级情感分类、方面级情感抽取与分类等多项基准测试上，SentiLARE均刷新了当时的最优结果。这项工作表明，将外部情感词典知识以结构化方式嵌入预训练过程，能够显著提升模型对情感语义的全局理解能力，为情感分析领域的预训练模型设计提供了新思路。
原文 arXiv:1911.02493;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.02493v3