Latent Relation Language Models
Hiroaki Hayashi†∗, Zecong Hu†∗, Chenyan Xiong‡, Graham Neubig† †Carnegie Mellon University, ‡Microsoft Research AI
Abstract
In this paper, we propose Latent Relation Language Models (LRLMs), a class of language models that parameterizes the joint distribution over the words in a document and the entities that occur therein via knowledge graph relations. This model has a number of attractive properties: it not only improves language modeling performance, but is also able to annotate the posterior probability of entity spans for a given text through relations. Experiments demonstrate empirical improvements over both a word-based baseline language model and a previous approach that incorporates knowledge graph information. Qualitative analysis further demonstrates the proposed model’s ability to learn to predict appropriate relations in context.**footnotetext: Equal contribution.
中文速览
语言模型在预测罕见词（尤其是命名实体）时往往力不从心，因为这类词在训练文本中出现太少，光靠统计规律学不好。本文提出的"潜在关系语言模型"（Latent Relation Language Model，LRLM）把知识图谱（Knowledge Graph，KG）中的实体关系引入语言模型：模型在生成每个词时，可以选择从固定词表里生成，也可以从知识图谱里匹配到某条关系后直接"抄"出对应的实体文本片段，选哪条路径被设计成潜在变量，通过前向-后向算法在训练时自动被边缘化掉。在Wikipedia文章上的实验表明，LRLM在困惑度（perplexity）上显著优于纯文本基线和此前引入KG信息的方法，并且对潜在关系的后验概率分析显示，模型自动学会了在正确上下文中激活对应的关系类型。这项工作的价值在于：它既提升了语言模型对罕见实体的建模能力，又顺带为实体链接（entity linking）提供了一个无需额外标注的概率框架。
原文 arXiv:1908.07690;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1908.07690v1