Structured Memory for Neural Turing Machines
Wei Zhang Yang Yu Bowen Zhou IBM Watson Alternative email for Wei Zhang:
Abstract
Neural Turing Machines (NTM) [2] contain memory component that simulates “working memroy” in the brain to store and retrieve information to ease simple algorithms learning. So far, only linearly organized memory is proposed, and during experiments, we observed that the model does not always converge, and overfits easily when handling certain tasks. We think memory component is key to some faulty behaviors of NTM, and better organization of memory component could help fight those problems. In this paper, we propose several different structures of memory for NTM, and we proved in experiments that two of our proposed structured-memory NTMs could lead to better convergence, in term of speed and prediction accuracy on copy task and associative recall task as in [2].
中文速览
神经图灵机（Neural Turing Machine，NTM）因为记忆组件组织方式单一，在训练时常出现收敛慢、不稳定乃至过拟合的问题。为此，作者提出了三种结构化记忆变体：NTM1 在原有受控记忆之外增加一块"隐藏记忆"做滑动平均平滑；NTM2 则用两个写头分别控制两层记忆并分层融合；NTM3 把多层 LSTM 控制器的中间层输出也分别连接到独立记忆块。在复制任务和关联回忆任务上的实验表明，NTM1 和 NTM2 比原版 NTM 收敛更快、异常损失点（过拟合信号）更少，NTM2 在关联回忆任务上甚至节省了约 13,000 步迭代；而 NTM3 因中间层引入额外噪声反而更不稳定。这项工作说明，为 NTM 设计合理的分层记忆结构是提升训练稳定性的有效思路，为后续在更复杂推理任务上改进记忆网络提供了参考方向。
原文 arXiv:1510.03931;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1510.03931v3