A Bayesian encourages dropout
Shin-ichi Maeda Graduate School of Informatics Kyoto University Yoshidahonmachi 36-1, Sakyo, Kyoto, Japan
Abstract
Dropout is one of the key techniques to prevent the learning from overfitting. It is explained that dropout works as a kind of modified L2 regularization. Here, we shed light on the dropout from Bayesian standpoint. Bayesian interpretation enables us to optimize the dropout rate, which is beneficial for learning of weight parameters and prediction after learning. The experiment result also encourages the optimization of the dropout.
中文速览
丢弃法（Dropout）是深度神经网络中防止过拟合的核心技巧，但此前学界对"丢弃率该设多少"缺乏理论依据，通常一律拍脑袋定为0.5。这篇论文从贝叶斯视角重新解读Dropout：把训练过程看作对所有子模型做贝叶斯模型平均的近似推断，把丢弃率对应的分布看作子模型的近似后验分布，从而将丢弃率的选取转化为一个可优化的学习问题——通过最大化边际似然的变分下界，在训练中自动估计最优丢弃率，而不是将其固定为超参数。实验结果表明，这种可自适应调整的"贝叶斯Dropout"在参数学习和预测精度上均优于固定丢弃率的标准做法。这项工作的意义在于，它不仅为Dropout提供了严格的理论根基，还给出了一套系统化调整丢弃率的方法，有望让大规模神经网络的训练更加稳健和高效。
原文 arXiv:1412.7003;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1412.7003v3