Dropout as data augmentation
Xavier Bouthillier Université de Montréal, Canada、Kishore Konda ∗ Goethe University Frankfurt, Germany \ANDPascal Vincent Université de Montréal, Canada and CIFAR、Roland Memisevic Université de Montréal, Canada Both authors contributed equally
Abstract
Dropout is typically interpreted as bagging a large number of models sharing parameters. We show that using dropout in a network can also be interpreted as a kind of data augmentation in the input space without domain knowledge. We present an approach to projecting the dropout noise within a network back into the input space, thereby generating augmented versions of the training data, and we show that training a deterministic network on the augmented samples yields similar results. Finally, we propose a new dropout noise scheme based on our observations and show that it improves dropout results without adding significant computational cost.
中文速览
把神经网络训练中常用的 Dropout 技术放到数据增强（data augmentation）的框架下重新审视，作者发现 Dropout 在隐层施加的随机噪声，其实等价于在输入空间对原始样本做了一种隐式的、无需领域知识的变换。他们提出一种方法，通过梯度下降将隐层噪声"反投影"回输入空间，从而生成与 Dropout 等效的增强样本，并在 MNIST 和 CIFAR-10 上验证了用这些确定性增强样本训练的网络与直接使用 Dropout 训练的网络精度相当。受此启发，他们进一步提出"随机 Dropout 水平"（random dropout）——将固定的 dropout 概率替换为每次从均匀分布中随机采样的概率，从而生成幅度更多样的变换——实验表明该方案在多种网络结构上均优于标准 Dropout，且几乎不带来额外计算开销。这项工作不仅为理解 Dropout 的正则化机制提供了新视角，也为设计更有效的噪声方案提供了可操作的思路。
原文 arXiv:1506.08700;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1506.08700v4