Training wide residual networks for deployment using a single bit for each weight
Mark D. McDonnell Computational Learning Systems Laboratory (cls-lab.org) School of Information Technology and Mathematical Sciences University of South Australia Mawson Lakes, SA 5095, AUSTRALIA This work was conducted, in part, during a hosted visit at the Institute for Neural Computation, University of California, San Diego, and in part, during a sabbatical period at Consilium Technology, Adelaide, Australia.
Abstract
For fast and energy-efficient deployment of trained deep neural networks on resource-constrained embedded hardware, each learned weight parameter should ideally be represented and stored using a single bit. Error-rates usually increase when this requirement is imposed. Here, we report large improvements in error rates on multiple datasets, for deep convolutional neural networks deployed with 1-bit-per-weight. Using wide residual networks as our main baseline, our approach simplifies existing methods that binarize weights by applying the sign function in training; we apply scaling factors for each layer with constant unlearned values equal to the layer-specific standard deviations used for initialization. For CIFAR-10, CIFAR-100 and ImageNet, and models with 1-bit-per-weight requiring less than 10 MB of parameter memory, we achieve error rates of 3.9%, 18.5% and 26.0% / 8.5% (Top-1 / Top-5) respectively. We also considered MNIST, SVHN and ImageNet32, achieving 1-bit-per-weight test results of 0.27%, 1.9%, and 41.3% / 19.1% respectively. For CIFAR, our error rates halve previously reported values, and are within about 1% of our error-rates for the same network with full-precision wei
中文速览
论文要解决的是：如何把深度卷积网络的每个权重压到1 bit，从而让模型更省内存、更省电、适合嵌入式设备，同时尽量不牺牲识别准确率。作者以宽残差网络为基础，训练时让权重在前向和反向传播中取正负1，但仍用全精度权重更新，并为每层加入根据初始化标准差设定的固定缩放因子，同时在容易过拟合的任务中固定批归一化参数、配合带热重启的学习率策略。结果显示，在CIFAR-10、CIFAR-100和ImageNet上，1 bit权重模型分别达到3.9%、18.5%和26.0%/8.5%的错误率，CIFAR数据上的错误率约为此前方法的一半，并且与全精度模型只差约1%。这说明极低精度权重不必以明显准确率下降为代价，经过简单训练改动就能得到接近全精度的模型，为低功耗芯片、传感器、机器人和物联网设备部署深度网络提供了更实用的路径。
原文 arXiv:1802.08530;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.08530v1