The State of Sparsity in Deep Neural Networks
Trevor Gale Erich Elsen Sara Hooker
Abstract
We rigorously evaluate three state-of-the-art techniques for inducing sparsity in deep neural networks on two large-scale learning tasks: Transformer trained on WMT 2014 English-to-German, and ResNet-50 trained on ImageNet. Across thousands of experiments, we demonstrate that complex techniques (Molchanov et al., 2017; Louizos et al., 2017b) shown to yield high compression rates on smaller datasets perform inconsistently, and that simple magnitude pruning approaches achieve comparable or better results. Based on insights from our experiments, we achieve a new state-of-the-art sparsity-accuracy trade-off for ResNet-50 using only magnitude pruning. Additionally, we repeat the experiments performed by Frankle & Carbin (2018) and Liu et al. (2018) at scale and show that unstructured sparse architectures learned through pruning cannot be trained from scratch to the same test set performance as a model trained with joint sparsification and optimization. Together, these results highlight the need for large-scale benchmarks in the field of model compression. We open-source our code, top performing model checkpoints, and results of all hyperparameter configurations to establish rigorous bas
中文速览
深度神经网络越做越大，如何在不明显损失精度的前提下大幅压缩模型参数量，是部署落地的核心难题。研究者们在ImageNet图像分类（ResNet-50）和WMT 2014英德机器翻译（Transformer）两个大规模任务上，系统对比了三种主流稀疏化（sparsification）方法：变分Dropout、L0正则化和幅度剪枝（magnitude pruning），共进行了逾4000次实验。结果显示，前两种在小数据集上表现亮眼的复杂方法在大规模任务上表现不稳定，而最简单的幅度剪枝反而能达到相当甚至更优的压缩-精度平衡，并借此刷新了ResNet-50的稀疏化最优记录。此外，实验还证明，通过剪枝得到的稀疏网络拓扑如果从头训练，无法复现边训练边剪枝所达到的精度，从而对"彩票假说"等相关观点给出了更清晰的大规模验证。这项工作的重要意义在于：它揭示了模型压缩领域长期依赖小规模基准测试的局限，并通过开源代码、模型权重和全部超参数结果，为后续研究提供了严格可靠的对照基线。
原文 arXiv:1902.09574;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.09574v1