Recurrent Dropout without Memory Loss
Stanislau Semeniuta Universität zu Lübeck Institut für Neuro- und Bioinformatik、Aliaksei Severyn Google Inc.、Erhardt Barth Universität zu Lübeck Institut für Neuro- und Bioinformatik Stanislau Semeniuta1 Aliaksei Severyn2 Erhardt Barth1 1Universität zu Lübeck, Institut für Neuro- und Bioinformatik 2Google Research
Abstract
This paper presents a novel approach to recurrent neural network (RNN) regularization. Differently from the widely adopted dropout method, which is applied to forward connections of feed-forward architectures or RNNs, we propose to drop neurons directly in recurrent connections in a way that does not cause loss of long-term memory. Our approach is as easy to implement and apply as the regular feed-forward dropout and we demonstrate its effectiveness for Long Short-Term Memory network, the most popular type of RNN cells. Our experiments on NLP benchmarks show consistent improvements even when combined with conventional feed-forward dropout.
中文速览
长短期记忆网络（LSTM）等循环神经网络容易过拟合，虽然dropout正则化在前馈网络中广泛有效，但直接把它用在循环连接上会破坏网络的长期记忆能力——因为反复缩放隐藏状态会让远处的历史信息在测试时指数级衰减消失。针对这一问题，作者提出只对每个时间步的"隐藏状态候选更新向量"施加dropout，而不是对隐藏状态本身动手，这样既能正则化循环权重，又不会损坏网络积累的长期记忆。在语言模型、命名实体识别等多个NLP基准上的实验表明，该方法不仅单独使用时效果优于已有的循环dropout方案，与传统前向dropout结合后仍能带来稳定提升。这项工作的意义在于填补了循环神经网络正则化的关键空白，提供了一种简单、无损记忆且即插即用的训练技巧。
原文 arXiv:1603.05118;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.05118v2