Improved Bayesian Compression
Marco Federici University of Amsterdam Ullrich University of Amsterdam Welling University of Amsterdam Canadian Institute for Advanced Research (CIFAR)
Abstract
Compression of Neural Networks (NN) has become a highly studied topic in recent years. The main reason for this is the demand for industrial scale usage of NNs such as deploying them on mobile devices, storing them efficiently, transmitting them via band-limited channels and most importantly doing inference at scale. There have been two proposals that show strong results, both using empirical Bayesian priors: (i) Ullrich et al. [2017] show impressive compression results by use of an adaptive Mixture of Gaussian prior on independent delta distributed weights. This idea has initially been proposed as Soft-Weight Sharing by Nowlan and Hinton [1992] but was never demonstrated to compress before. (ii) Equivalently, Molchanov et al. [2017] use Variational Dropout [Kingma et al., 2015] to prune out independent Gaussian posterior weights with high uncertainties. To achieve high pruning rates the authors refined the originally proposed approximations to the KL-divergence and a different parametrization to increase the stability of the training procedure. In this work, we propose to join these two somewhat orthogonal compression schemes since (ii) seems to prune out more weights but does not
中文速览
神经网络的模型压缩面临两个核心挑战：一是剪枝（去掉不重要的权重），二是量化（把权重聚类到少数离散值），而现有方法往往只擅长其中一个。这篇论文把两种经典贝叶斯压缩方案——基于变分Dropout（Variational Dropout）的稀疏剪枝和基于软权重共享（Soft Weight Sharing）的高斯混合量化——整合进一个统一的变分贝叶斯框架：对权重本身施加对数均匀先验来诱导稀疏性，同时对权重中心施加可学习的高斯混合先验来驱动聚类，两者联合优化一个变分下界。训练结束后，再依次经过权重聚类、结构化剪枝、稀疏行压缩编码和哈夫曼编码四步流水线，在MNIST的LeNet300-100和LeNet5两种架构上均取得了当时最优的压缩比，且精度损失极小。这项工作的意义在于证明了剪枝与量化并非非此即彼，将二者统一建模可以显著放大各自的压缩收益，为在移动端等资源受限场景下高效部署神经网络提供了更强的理论与实践基础。
原文 arXiv:1711.06494;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.06494v2