Variational Dropout Sparsifies Deep Neural Networks
Dmitry Molchanov Arsenii Ashukha Dmitry Vetrov
Abstract
We explore a recently proposed Variational Dropout technique that provided an elegant Bayesian interpretation to Gaussian Dropout. We extend Variational Dropout to the case when dropout rates are unbounded, propose a way to reduce the variance of the gradient estimator and report first experimental results with individual dropout rates per weight. Interestingly, it leads to extremely sparse solutions both in fully-connected and convolutional layers. This effect is similar to automatic relevance determination effect in empirical Bayes but has a number of advantages. We reduce the number of parameters up to $280$ times on LeNet architectures and up to $68$ times on VGG-like networks with a negligible decrease of accuracy.
中文速览
深度神经网络参数动辄数亿，既容易过拟合又占用大量计算资源，如何在保持精度的同时大幅压缩模型是个核心难题。这篇文章在"变分Dropout（Variational Dropout）"的贝叶斯框架基础上，为网络的每个权重单独学习一个dropout率，并通过"加性噪声重参数化"技巧大幅降低梯度估计的方差、同时引入更精确的KL散度近似，从而使dropout率可以无限增大——当某个权重的dropout率趋向无穷时，该权重实际上被自动置零，模型由此天然获得极度稀疏性。实验表明，在LeNet架构上参数量最多压缩280倍、在VGG类网络上压缩68倍，而精度几乎没有损失，且该方法不会像普通Dropout那样在随机标签数据上过拟合。这一结果意味着无需手动剪枝或额外正则化，仅靠贝叶斯训练本身就能同时完成模型压缩与正则化，对在资源受限场景下部署大型神经网络具有重要实用价值。
原文 arXiv:1701.05369;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1701.05369v3