Adaptive Semiparametric Language Models
Dani Yogatama, Cyprien de Masson d’Autume, Lingpeng Kong DeepMind London, United Kingdom
Abstract
We present a language model that combines a large parametric neural network (i.e., a transformer) with a non-parametric episodic memory component in an integrated architecture. Our model uses extended short-term context by caching local hidden states—similar to transformer-XL—and global long-term memory by retrieving a set of nearest neighbor tokens at each timestep. We design a gating function to adaptively combine multiple information sources to make a prediction. This mechanism allows the model to use either local context, short-term memory, or long-term memory (or any combination of them) on an ad hoc basis depending on the context. Experiments on word-based and character-based language modeling datasets demonstrate the efficacy of our proposed method compared to strong baselines.
中文速览
传统语言模型把所有知识都塞进一个大型神经网络的参数里，既难以处理超长上下文，又无法灵活调取历史信息。这篇论文提出了名为 Spalm 的语言模型架构，将一个大型 Transformer 参数网络与两种外部记忆模块整合在一起：一个类似 Transformer-XL 的短期记忆（缓存近期隐藏状态），以及一个基于 k 近邻检索（k-nearest neighbor retrieval）的长期情节记忆（episodic memory）数据库。关键创新在于设计了一个上下文相关的门控机制（gating mechanism），让模型在每个时间步根据当前语境自适应地决定该更多依赖局部信息、短期记忆还是长期记忆，而不是像 kNN-LM 那样用一个固定插值权重统一处理所有词。在基于词和基于字符的语言建模基准上，Spalm 均优于 Transformer-XL 和 kNN-LM 等强基线，同时分析实验也验证了模型确实学会了在"简单"预测时依赖局部上下文、在"困难"预测时调用长期记忆；这一模块化设计思路不仅提升了语言模型性能，也为将人类记忆系统的分工理念引入序列建模提供了可行范式。
原文 arXiv:2102.02557;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.02557v1