A Neural Knowledge Language Model
Sungjin Ahn Heeyoul Choi Tanel Pärnamaa Yoshua Bengio
Abstract
Current language models have significant limitation in the ability to encode and decode factual knowledge. This is mainly because they acquire such knowledge from statistical co-occurrences although most of the knowledge words are rarely observed. In this paper, we propose a Neural Knowledge Language Model (NKLM) which combines symbolic knowledge provided by the knowledge graph with the RNN language model. By predicting whether the word to generate has an underlying fact or not, the model can generate such knowledge-related words by copying from the description of the predicted fact. In experiments, we show that the NKLM significantly improves the performance while generating a much smaller number of unknown words.
中文速览
传统语言模型依靠语料库中的统计共现规律来学习知识，但人名、地名、年份等知识性词汇本身出现频率低，导致模型既难以理解也难以生成这类词。作者提出了神经知识语言模型（Neural Knowledge Language Model，NKLM），将知识图谱（knowledge graph）中的结构化事实引入循环神经网络语言模型，让模型在每一步生成词时先判断该词是否与某条事实相关，若相关则直接从该事实的文字描述中"复制"对应词，从而绕开词表限制、解决生成罕见词和未知词的难题。实验结果显示，NKLM在困惑度（perplexity）等指标上显著优于基线模型，生成的未知词数量大幅减少，并且因为事实存储在可随时更新的外部知识库中，无需重新训练模型就能适应新知识。这项工作为将符号化知识与神经语言模型深度结合提供了一条可行路径，对问答、对话等需要大量事实推理的应用具有重要意义。
原文 arXiv:1608.00318;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.00318v2