Batch Normalized Recurrent Neural Networks
César Laurent Université de Montréal、Gabriel Pereyra ∗ University of Southern California \ANDPhilémon Brakel Université de Montréal、Ying Zhang Université de Montréal、Yoshua Bengio Université de Montréal Equal contributionCIFAR Senior Fellow
Abstract
Recurrent Neural Networks (RNNs) are powerful models for sequential data that have the potential to learn long-term dependencies. However, they are computationally expensive to train and difficult to parallelize. Recent work has shown that normalizing intermediate representations of neural networks can significantly improve convergence rates in feedforward neural networks [1]. In particular, batch normalization, which uses mini-batch statistics to standardize features, was shown to significantly reduce training time. In this paper, we show that applying batch normalization to the hidden-to-hidden transitions of our RNNs doesn’t help the training procedure. We also show that when applied to the input-to-hidden transitions, batch normalization can lead to a faster convergence of the training criterion but doesn’t seem to improve the generalization performance on both our language modelling and speech recognition tasks. All in all, applying batch normalization to RNNs turns out to be more challenging than applying it to feedforward networks, but certain variants of it can still be beneficial.
中文速览
循环神经网络（RNN）在序列建模上很强大，但训练慢、难并行，人们自然想到把已在前馈网络上大放异彩的批归一化（Batch Normalization，BN）搬过来提速。作者系统地试验了把BN应用到RNN不同位置的效果，发现把BN加在隐层到隐层的循环连接上根本没有帮助，但只加在输入到隐层的连接上，确实能让训练损失下降更快——在语音识别和语言模型两个任务上都观察到了约两倍的收敛加速。不过代价是过拟合加剧，最终的泛化性能并不比基线模型更好，说明BN在RNN中能改善优化条件，却无法像在卷积网络里那样同时带来正则化收益。这项工作厘清了BN迁移到RNN的边界和局限，为后续针对循环结构的归一化方法（如Layer Norm）指明了方向。
原文 arXiv:1510.01378;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1510.01378v1