Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning
Tao Shen University of Technology Sydney、Yi Mao, Pengcheng He Microsoft Dynamics 365 AI \ANDGuodong Long University of Technology Sydney、Adam Trischler Microsoft Research, Montréal、Weizhu Chen Microsoft Dynamics 365 AI Work done while the author was an intern at Microsoft.
Abstract
In this work, we aim at equipping pre-trained language models with structured knowledge. We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme that exploits relational knowledge underlying the text. This is fulfilled by using a linked knowledge graph to select informative entities and then masking their mentions. In addition we use knowledge graphs to obtain distractors for the masked entities, and propose a novel distractor-suppressed ranking objective which is optimized jointly with masked language model. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training, to inject language models with structured knowledge via learning from raw text. It is more efficient than retrieval-based methods that perform entity linking and integration during finetuning and inference, and generalizes more effectively than the methods that directly learn from concatenated graph triples. Experiments show that our proposed model achieves improved performance on five benchmark datasets, including question an
中文速览
知识图谱中蕴藏着大量结构化关系知识，但如何把这些知识有效注入预训练语言模型（pre-trained language model）一直是个难题——现有方法要么在推理阶段实时检索图谱、代价高昂，要么直接在三元组文本上微调、泛化性差。本文提出了一种名为 GLM（Graph-guided Masked Language Model）的方法，仅在预训练阶段借助知识图谱的隐式引导来增强语言模型：一方面设计了基于知识图谱的实体掩码策略，优先遮盖那些"有据可推却非泛词"的实体，让模型从原始文本中主动学习关系知识；另一方面引入"抑制干扰项排序"（distractor-suppressed ranking）目标，用知识图谱采样的负例实体作为干扰项，与掩码语言模型联合训练，迫使模型区分真实实体与相似干扰项。在问答和知识库补全共五个基准测试上，GLM 达到了最优或有竞争力的表现，且推理时无需任何图谱检索操作，兼顾了效率与泛化能力。
原文 arXiv:2004.14224;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.14224v1