Inducing Relational Knowledge from BERT
Zied Bouraoui CRIL - CNRS、Univ Artois, France、Jose Camacho-Collados Cardiff University, UK、Steven Schockaert Cardiff University, UK
Abstract
One of the most remarkable properties of word embeddings is the fact that they capture certain types of semantic and syntactic relationships. Recently, pre-trained language models such as BERT have achieved groundbreaking results across a wide range of Natural Language Processing tasks. However, it is unclear to what extent such models capture relational knowledge beyond what is already captured by standard word embeddings. To explore this question, we propose a methodology for distilling relational knowledge from a pre-trained language model. Starting from a few seed instances of a given relation, we first use a large text corpus to find sentences that are likely to express this relation. We then use a subset of these extracted sentences as templates. Finally, we fine-tune a language model to predict whether a given word pair is likely to be an instance of some relation, when given an instantiated template for that relation as input.
中文速览
词向量（word embeddings）能捕捉词语之间的语义关系已经人尽皆知，但像BERT这样的预训练语言模型究竟比词向量多知道多少"关系知识"，目前还不清楚。研究者针对"关系归纳"任务——即给定少数几个示例词对，推断出同一关系的更多实例——提出了一套从预训练语言模型中蒸馏关系知识的方法：先从大规模文本语料库中找出提及示例词对的句子，再用BERT的填空能力筛选出真正能表达目标关系的触发句模板，最后用这些模板微调语言模型，使其判断任意新词对是否属于该关系。关键之处在于，对新词对的预测完全不依赖语料库中是否出现过这对词，而是纯粹依赖BERT内化的关系知识，语料库只用于寻找高质量的触发句。这项工作揭示了如何系统性地"问对问题"才能从语言模型中有效提取关系知识，对知识图谱构建、常识推理等下游应用具有重要参考价值。
原文 arXiv:1911.12753;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.12753v1