Model Compression via Distillation and Quantization
Antonio Polino ETH Zürich、Razvan Pascanu Google DeepMind、Dan Alistarh IST Austria
Abstract
Deep neural networks (DNNs) continue to make significant advances, solving tasks from image classification to translation or reinforcement learning. One aspect of the field receiving considerable attention is efficiently executing deep models in resource-constrained environments, such as mobile or embedded devices. This paper focuses on this problem, and proposes two new compression methods, which jointly leverage weight quantization and distillation of larger networks, called “teachers,” into compressed “student” networks. The first method we propose is called quantized distillation and leverages distillation during the training process, by incorporating distillation loss, expressed with respect to the teacher network, into the training of a smaller student network whose weights are quantized to a limited set of levels. The second method, differentiable quantization, optimizes the location of quantization points through stochastic gradient descent, to better fit the behavior of the teacher model. We validate both methods through experiments on convolutional and recurrent architectures. We show that quantized shallow students can reach similar accuracy levels to state-of-the-art fu
中文速览
深度神经网络在图像分类、翻译等任务上表现出色，但模型体积大、计算量高，难以直接部署到手机或嵌入式设备上。这篇论文提出了两种结合"知识蒸馏（knowledge distillation）"与"权重量化（weight quantization）"的压缩方法：第一种叫"量化蒸馏（quantized distillation）"，在训练较浅、权重被压缩到有限比特的"学生"网络时，同步引入大型"教师"网络的软标签作为额外监督信号，让学生网络学得更准；第二种叫"可微量化（differentiable quantization）"，通过随机梯度下降直接优化量化点的位置，使量化结果更贴合教师模型的行为。在图像分类（CIFAR、ImageNet）和机器翻译（WMT）等多个基准上的实验表明，量化后的浅层学生网络能达到与全精度深层教师网络相当的准确率，同时实现近十倍的模型压缩，推理速度也随层数减少近线性提升。这项工作的价值在于，它让资源受限的边缘设备也能享受到在高性能服务器上训练出的顶尖模型精度，为"大模型能力下沉到小设备"提供了一条切实可行的技术路径。
原文 arXiv:1802.05668;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.05668v1