To prune, or not to prune: exploring the efficacy of pruning for model compression
Michael H. Zhu Department of Computer Science Stanford University Stanford, CA 94305 \AndSuyog Gupta Google Inc. 1600 Amphitheatre Pkwy Mountain View, CA 94043 This research was done while the author was an intern at Google.
Abstract
Model pruning seeks to induce sparsity in a deep neural network’s various connection matrices, thereby reducing the number of nonzero-valued parameters in the model. Recent reports (Han et al., 2015a; Narang et al., 2017) prune deep networks at the cost of only a marginal loss in accuracy and achieve a sizable reduction in model size. This hints at the possibility that the baseline models in these experiments are perhaps severely over-parameterized at the outset and a viable alternative for model compression might be to simply reduce the number of hidden units while maintaining the model’s dense connection structure, exposing a similar trade-off in model size and accuracy. We investigate these two distinct paths for model compression within the context of energy-efficient inference in resource-constrained environments and propose a new gradual pruning technique that is simple and straightforward to apply across a variety of models/datasets with minimal tuning and can be seamlessly incorporated within the training process. We compare the accuracy of large, but pruned models (large-sparse) and their smaller, but dense (small-dense) counterparts with identical memory footprint. Across
中文速览
神经网络剪枝（model pruning）可以把模型里大量参数强制归零，从而压缩模型体积，但一个更直接的问题是：既然原始模型可能严重冗余，为什么不干脆训练一个参数量相当的小型密集模型，而非要费劲剪枝？论文系统地比较了"大模型剪枝后得到的稀疏模型（large-sparse）"与"参数量相当的小型密集模型（small-dense）"在相同内存占用下的精度表现，并提出了一种简单易用的渐进式剪枝算法——通过预设稀疏度时间表自动调整剪枝幅度，无需为每层手动设定阈值，能平滑融入常规训练流程。在图像分类（InceptionV3、MobileNets）、语言模型（堆叠LSTM）和机器翻译（seq2seq LSTM）等多类架构上的实验一致表明，large-sparse模型的精度明显优于内存占用相同的small-dense模型，最高可在精度几乎不下降的前提下将非零参数量压缩至原来的十分之一。这一结论对于在手机、智能摄像头等资源受限设备上部署高精度神经网络推理具有重要的指导意义。
原文 arXiv:1710.01878;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.01878v2