Scaling Memory-Augmented Neural Networks with Sparse Reads and Writes
Jack W Rae111These authors contributed equally. jwrae、Jonathan J Hunt111These authors contributed equally. jjhunt、Tim Harley tharley、Ivo Danihelka danihelka、Andrew Senior andrewsenior、Greg Wayne gregwayne、Alex Graves gravesa、Timothy P Lillicrap countzero Google DeepMind @google.com
Abstract
Neural networks augmented with external memory have the ability to learn algorithmic solutions to complex tasks. These models appear promising for applications such as language modeling and machine translation. However, they scale poorly in both space and time as the amount of memory grows — limiting their applicability to real-world domains. Here, we present an end-to-end differentiable memory access scheme, which we call Sparse Access Memory (SAM), that retains the representational power of the original approaches whilst training efficiently with very large memories. We show that SAM achieves asymptotic lower bounds in space and time complexity, and find that an implementation runs $1,\!000\times$ faster and with $3,\!000\times$ less physical memory than non-sparse models. SAM learns with comparable data efficiency to existing models on a range of synthetic tasks and one-shot Omniglot character recognition, and can scale to tasks requiring $100,\!000$ s of time steps and memories. As well, we show how our approach can be adapted for models that maintain temporal associations between memories, as with the recently introduced Differentiable Neural Computer.
中文速览
神经网络配上外部记忆库（external memory）虽然能学会解复杂任务，但随着记忆容量增大，计算时间和物理内存的消耗会线性甚至更快地爆炸式增长，根本没法用在大规模现实场景中。为此，研究者提出了"稀疏访问记忆"（Sparse Access Memory，SAM）方案：每次读写只操作少量关键记忆槽，再借助近似最近邻（approximate nearest neighbor）索引结构实现对数级别的快速检索，同时在反向传播时只追踪稀疏修改而非缓存整个记忆矩阵，从而把时间和空间复杂度都压到了理论最优。实验表明，SAM 比经典神经图灵机（Neural Turing Machine）快约 1000 倍、节省约 3000 倍物理内存，同时在合成算法任务、问答推理和 Omniglot 单样本字符识别上保持了几乎相同的学习效率，还能扩展到需要十万步时序和超大记忆的任务。这项工作首次让带外部记忆的神经网络在规模和效率上达到实用门槛，为语言建模、机器翻译等真实应用打开了大门。
原文 arXiv:1610.09027;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1610.09027v1