Sharp Nearby, Fuzzy Far Away: How Neural Language Models Use Context
Urvashi Khandelwal, He He, Peng Qi, Dan Jurafsky Computer Science Department Stanford University
Abstract
We know very little about how neural language models (LM) use prior linguistic context. In this paper, we investigate the role of context in an LSTM LM, through ablation studies. Specifically, we analyze the increase in perplexity when prior context words are shuffled, replaced, or dropped. On two standard datasets, Penn Treebank and WikiText-2, we find that the model is capable of using about 200 tokens of context on average, but sharply distinguishes nearby context (recent 50 tokens) from the distant history. The model is highly sensitive to the order of words within the most recent sentence, but ignores word order in the long-range context (beyond 50 tokens), suggesting the distant past is modeled only as a rough semantic field or topic. We further find that the neural caching model Grave et al. (2017b) especially helps the LSTM to copy words from within this distant context. Overall, our analysis not only provides a better understanding of how neural LMs use their context, but also sheds light on recent success from cache-based models.
中文速览
用消融实验（ablation study）来拆解LSTM语言模型（language model）到底在用多长、用什么样的上下文，是理解神经语言模型工作机制的关键一步。研究者在Penn Treebank和WikiText-2两个标准数据集上，通过在测试时对输入上下文进行截断、打乱顺序、随机替换等扰动，系统地测量了模型性能的变化。结果发现：LSTM平均能有效利用约200个词的历史信息，但对"近处"（约50词以内）和"远处"上下文的处理方式截然不同——近处上下文中词序至关重要，而50词之外的远距离上下文中词序几乎无影响，模型只保留了一种粗粒度的语义场或话题信息。此外，神经缓存机制（neural cache）的作用主要体现在帮助模型从这段远距离上下文中"复制"出现过的词，从而解释了缓存模型近年来取得成功的原因。这项工作首次从上下文长度与结构两个维度系统剖析了LSTM语言模型的信息利用机制，为理解和改进神经语言模型提供了重要的实证基础。
原文 arXiv:1805.04623;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1805.04623v1