E.T.: Entity-Transformers Coreference augmented Neural Language Model for richer mention representations via Entity-Transformer blocks
Nikolaos Stylianou Aristotle University of Thessaloniki School of Informatics Greece、Ioannis Vlahavas Aristotle University of Thessaloniki School of Informatics Greece
Abstract
In the last decade, the field of Neural Language Modelling has witnessed enormous changes, with the development of novel models through the use of Transformer architectures. However, even these models struggle to model long sequences due to memory constraints and increasing computational complexity. Coreference annotations over the training data can provide context far beyond the modelling limitations of such language models. In this paper we present an extension over the Transformer-block architecture used in neural language models, specifically in GPT2, in order to incorporate entity annotations during training. Our model, GPT2E, extends the Transformer layers architecture of GPT2 to Entity-Transformers, an architecture designed to handle coreference information when present. To that end, we achieve richer representations for entity mentions, with insignificant training cost. We show the comparative model performance between GPT2 and GPT2E in terms of Perplexity on the CoNLL 2012 and LAMBADA datasets as well as the key differences in the entity representations and their effects in downstream tasks such as Named Entity Recognition. Furthermore, our approach can be adopted by the m
中文速览
语言模型在处理长文本时面临"记忆缺失"的老难题——当一个实体在几百个词之前首次出现，模型往往已经忘了它是谁。为解决这一问题，研究者在 GPT-2 的 Transformer 块基础上设计了一种新架构 Entity-Transformer（实体-变换器），并将其整合为 GPT2E 模型：通过在每个 Transformer 块末尾引入一个额外的"实体注意力层"，把文本中共指（coreference）标注所识别的实体向量作为键值直接注入注意力计算，使模型能够在整个文档范围内持续追踪和更新同一实体的表示。在 CoNLL-2012 和 LAMBADA 两个数据集上的实验表明，GPT2E 与基线 GPT-2 的困惑度（Perplexity）相近，但在以专有名词和普通名词为预测目标的 LAMBADA 任务上略胜一筹，且同一实体不同提及的向量表示更加接近，下游命名实体识别（NER）任务的性能也有所提升。这项工作的意义在于：它以几乎可忽略的额外训练代价，为主流 Transformer 语言模型提供了一种即插即用的实体感知改造方案，为突破模型上下文窗口限制、构建更具"人物记忆"的语言模型开辟了新思路。
原文 arXiv:2011.05431;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.05431v1