E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT
Nina Poerner∗† and Ulli Waltinger† and Hinrich Schütze∗ ∗Center for Information and Language Processing, LMU Munich, Germany †Corporate Technology Machine Intelligence (MIC-DE), Siemens AG Munich, Germany |
Abstract
We present a novel way of injecting factual knowledge about entities into the pretrained BERT model Devlin et al. (2019): We align Wikipedia2Vec entity vectors Yamada et al. (2016) with BERT’s native wordpiece vector space and use the aligned entity vectors as if they were wordpiece vectors. The resulting entity-enhanced version of BERT (called E-BERT) is similar in spirit to ERNIE Zhang et al. (2019) and KnowBert Peters et al. (2019), but it requires no expensive further pretraining of the BERT encoder. We evaluate E-BERT on unsupervised question answering (QA), supervised relation classification (RC) and entity linking (EL). On all three tasks, E-BERT outperforms BERT and other baselines. We also show quantitatively that the original BERT model is overly reliant on the surface form of entity names (e.g., guessing that someone with an Italian-sounding name speaks Italian), and that E-BERT mitigates this problem.
中文速览
把实体知识注入BERT一直需要对编码器做额外预训练，成本高昂。研究者提出了E-BERT：先用线性变换把Wikipedia2Vec的实体向量对齐到BERT原生词片（wordpiece）向量空间，再把对齐后的实体向量像普通词片向量一样直接喂给BERT，整个过程无需改动编码器或做任何额外预训练。在无监督问答（LAMA基准）、关系分类和实体链接三项任务上，E-BERT均超过了原版BERT、ERNIE和KnowBert，而训练成本仅需约6小时CPU时间，远低于竞品。此外，研究者还发现原版BERT严重依赖实体名称的表面形式来猜答案（如看到法语名字就猜"法语"），并为此构建了LAMA-UHN这一更严格的测试集来剔除这类"捷径"问题，证明E-BERT在真正需要事实记忆的场景下优势更为突出。
原文 arXiv:1911.03681;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.03681v2