Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Sergey Ioffe Google Inc., Christian Szegedy Google Inc.,
Abstract
Training Deep Neural Networks is complicated by the fact that the distribution of each layer’s inputs changes during training, as the parameters of the previous layers change. This slows down the training by requiring lower learning rates and careful parameter initialization, and makes it notoriously hard to train models with saturating nonlinearities. We refer to this phenomenon as internal covariate shift, and address the problem by normalizing layer inputs. Our method draws its strength from making normalization a part of the model architecture and performing the normalization for each training mini-batch. Batch Normalization allows us to use much higher learning rates and be less careful about initialization. It also acts as a regularizer, in some cases eliminating the need for Dropout. Applied to a state-of-the-art image classification model, Batch Normalization achieves the same accuracy with 14 times fewer training steps, and beats the original model by a significant margin. Using an ensemble of batch-normalized networks, we improve upon the best published result on ImageNet classification: reaching 4.9% top-5 validation error (and 4.8% test error), exceeding the accuracy of
中文速览
深度神经网络在训练时面临一个棘手问题：随着前层参数不断更新，后层接收到的输入分布也在持续变化，这种"内部协变量偏移"（internal covariate shift）会迫使网络只能用很小的学习率缓慢训练，还容易陷入饱和非线性的梯度消失困境。为此，论文提出了批归一化（Batch Normalization，BN）——在每一层的激活值进入下一层之前，利用当前小批量数据的均值和方差对其进行标准化，再通过一对可学习的缩放与平移参数恢复网络的表达能力，从而把归一化操作直接嵌入模型架构并参与反向传播。实验表明，加入BN后可以使用大得多的学习率，对参数初始化的要求也大幅降低，同时还兼具正则化效果，在某些情况下可以去掉Dropout；应用到当时最先进的ImageNet分类模型上，BN只需原来约7%的训练步数就能达到相同精度，集成多个BN网络后更以4.8%的top-5测试错误率首次超越人类标注水平。这项工作从根本上改变了深度网络的训练范式，成为此后几乎所有深度学习架构的标配组件。
原文 arXiv:1502.03167;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1502.03167v3