Adjusting for Dropout Variance in Batch Normalization and Weight Initialization
Dan Hendrycks University of Chicago、Kevin Gimpel Toyota Technological Institute at Chicago Work done while the author was at TTIC. Code available at github.com/hendrycks/init
Abstract
We show how to adjust for the variance introduced by dropout with corrections to weight initialization and Batch Normalization, yielding higher accuracy. Though dropout can preserve the expected input to a neuron between train and test, the variance of the input differs. We thus propose a new weight initialization by correcting for the influence of dropout rates and an arbitrary nonlinearity’s influence on variance through simple corrective scalars. Since Batch Normalization trained with dropout estimates the variance of a layer’s incoming distribution with some inputs dropped, the variance also differs between train and test. After training a network with Batch Normalization and dropout, we simply update Batch Normalization’s variance moving averages with dropout off and obtain state of the art on CIFAR-10 and CIFAR-100 without data augmentation.
中文速览
深度神经网络在训练时使用 Dropout 会向每层的输入分布引入额外的方差，但现有的权重初始化方法和批归一化（Batch Normalization）都没有专门处理这一问题。针对此，作者提出了两个修正方案：一是推导出一种新的权重初始化公式，通过引入与 Dropout 保留率和激活函数特性相关的校正系数，使前向传播和反向传播过程中各层的信号方差保持稳定；二是指出批归一化在训练时估计的方差统计量是在 Dropout 激活状态下得到的，与测试时（Dropout 关闭）的实际分布不匹配，因此在训练结束后只需关闭 Dropout、重新对训练数据做一次前向传播以更新批归一化的方差滑动平均值即可修正这一偏差。实验表明，新初始化方法在 MNIST 和 CIFAR-10 上比 He 初始化收敛更快、精度更高，而方差重估技术应用于 DenseNet 后在不使用数据增强的条件下刷新了 CIFAR-10 和 CIFAR-100 的最优结果。这项工作揭示了 Dropout 与方差控制机制之间长期被忽视的不匹配问题，所提方案简单通用，无需修改训练流程即可带来显著的性能提升。
原文 arXiv:1607.02488;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1607.02488v2