Associative Long Short-Term Memory
Ivo Danihelka Greg Wayne Benigno Uria Nal Kalchbrenner Alex Graves Google DeepMind
Abstract
We investigate a new method to augment recurrent neural networks with extra memory without increasing the number of network parameters. The system has an associative memory based on complex-valued vectors and is closely related to Holographic Reduced Representations and Long Short-Term Memory networks. Holographic Reduced Representations have limited capacity: as they store more information, each retrieval becomes noisier due to interference. Our system in contrast creates redundant copies of stored information, which enables retrieval with reduced noise. Experiments demonstrate faster learning on multiple memorization tasks.
中文速览
用复数向量构建联想记忆、并让循环神经网络学会"按内容检索"，一直是序列建模的难题。研究者将全息简化表示（Holographic Reduced Representations，HRR）与长短期记忆网络（LSTM）结合，提出了"联想式LSTM"（Associative LSTM）——用复数向量的逐元素乘法实现键值绑定，再通过存储多份随机置换的冗余副本来压制检索噪声，整个扩展记忆模块不需要增加任何网络参数。理论分析和ImageNet图像编码实验均证实，检索误差与存储条目数成正比、与副本数成反比，冗余存储能有效提升记忆容量。在片段复制、变量赋值、XML建模等多项记忆基准任务上，联想式LSTM的学习速度和准确率均明显优于标准LSTM及酉矩阵RNN等基线，表明这种无参数开销的外部联想记忆机制是增强序列模型记忆能力的实用途径。
原文 arXiv:1602.03032;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1602.03032v2