Efficient Nearest Neighbor Language Models
Junxian He†, Graham Neubig†, Taylor Berg-Kirkpatrick‡ †Language Technologies Institute, Carnegie Mellon University ‡Department of Computer Science and Engineering, University of California San Diego {junxianh,
Abstract
Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore, which allows them to learn through explicitly memorizing the training datapoints. While effective, these models often require retrieval from a large datastore at test time, significantly increasing the inference overhead and thus limiting the deployment of non-parametric NLMs in practical applications. In this paper, we take the recently proposed $k$ -nearest neighbors language model (Khandelwal et al., 2020) as an example, exploring methods to improve its efficiency along various dimensions. Experiments on the standard WikiText-103 benchmark and domain-adaptation datasets show that our methods are able to achieve up to a 6x speed-up in inference speed while retaining comparable performance. The empirical analysis we present may provide guidelines for future research seeking to develop or deploy more efficient non-parametric NLMs.111Code is available at https://github.com/jxhe/efficient-knnlm.
中文速览
kNN语言模型（kNN-LM）通过在推理时从一个巨型外部数据库中检索最近邻上下文来提升预测质量，但这个检索步骤会让推理速度比普通神经语言模型慢10到30倍，严重制约了它的实际部署。针对这一瓶颈，作者从三个维度入手：训练一个轻量网络自适应地跳过不必要的检索操作、对数据库进行聚类或过滤剪枝以缩小规模、以及对存储向量进行降维压缩。在WikiText-103和法律领域自适应数据集上的实验表明，综合使用这些方法可以在困惑度基本不变的前提下实现高达6倍的推理加速。这项工作为如何在性能与效率之间取得平衡提供了可量化的经验指导，有助于推动非参数语言模型走向实际应用。
原文 arXiv:2109.04212;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.04212v3