Fast Parametric Learning with Activation Memorization
Jack W Rae Chris Dyer Peter Dayan Timothy P Lillicrap
Abstract
Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non-parametric model which stores recent activations and class labels into an external memory. We explore a simplified architecture where we treat a subset of the model parameters as fast memory stores. This can help retain information over longer time intervals than a traditional memory, and does not require additional space or compute. In the case of image classification, we display faster binding of novel classes on an Omniglot image curriculum task. We also show improved performance for word-based language models on news reports (GigaWord), books (Project Gutenberg) and Wikipedia articles (WikiText-103) — the latter achieving a state-of-the-art perplexity of $29.2$ .
中文速览
用反向传播训练的神经网络在识别"见过次数极少的类别"时表现很差，而在语言建模这类绝大多数词汇都属于低频词的任务中，这个问题尤为突出。为此，研究者提出了一种叫做"Hebbian Softmax"的方法：在网络的softmax输出层引入一套双速学习规则——对于刚刚出现过很少次数的类别，直接把网络当前的隐藏激活向量写入（累积平滑到）对应的权重参数中，相当于把参数本身当作一块快速更新的压缩记忆；而随着该类别出现次数增多，学习规则逐渐过渡到普通的梯度下降，让表示慢慢稳定下来。与外部缓存记忆（cache）相比，这种做法不需要额外的存储空间，稀有类别的记忆也不会因缓存窗口滑出而失效。实验表明，该方法在Omniglot手写字符任务上能更快绑定新类别，在GigaWord、Project Gutenberg和WikiText-103三个语言建模基准上均降低了困惑度（perplexity），其中WikiText-103达到了当时最优的29.2，证明了这一简单改动对低频词建模有切实且持久的收益。
原文 arXiv:1803.10049;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.10049v1