Training wide residual networks for deployment using a single bit for each weight
Mark D. McDonnell Computational Learning Systems Laboratory (cls-lab.org) School of Information Technology and Mathematical Sciences University of South Australia Mawson Lakes, SA 5095, AUSTRALIA This work was conducted, in part, during a hosted visit at the Institute for Neural Computation, University of California, San Diego, and in part, during a sabbatical period at Consilium Technology, Adelaide, Australia.
Abstract
For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight. Using wide residual networks as our main baseline, our approach simplifies existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-specific standard deviations used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% respectively. For CIFAR, our error rates halve previously reported values, and are within about 1% of our error-rates for the same network with full-precision wei
中文速览
把用于边缘硬件部署的深度卷积神经网络每个权重压缩到仅1个比特，通常会导致精度大幅下降，这篇论文正面攻克了这一难题。作者的核心方法出奇简单：训练时对每层权重取符号函数（+1/-1），同时乘以一个固定不可学习的缩放因子（等于He初始化对应的标准差），从而让激活值和梯度的方差在整个网络中保持稳定，避免了以往方法收敛慢、精度低的问题；此外，他们还发现对容易过拟合的网络去掉批归一化的可学习缩放和偏移参数能进一步降低误差，并配合余弦退火热重启学习率加速训练。最终在CIFAR-10/100上，1比特权重网络的错误率相比此前最优结果直接减半，分别达到3.9%和18.5%，与全精度网络的差距缩小到约1%，ImageNet上也取得26.0% Top-1的成绩，而模型参数存储不超过10 MB。这项工作证明极度压缩的网络完全可以逼近全精度精度，为在物联网、机器人等资源受限设备上实用化部署深度学习铺平了道路。
原文 arXiv:1802.08530;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.08530v1