DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, Yuheng Zou Megvii Inc. {zsc, wyx, nzk, zxy, wenhe,
Abstract
We propose DoReFa-Net, a method to train convolutional neural networks that have low bitwidth weights and activations using low bitwidth parameter gradients. In particular, during backward pass, parameter gradients are stochastically quantized to low bitwidth numbers before being propagated to convolutional layers. As convolutions during forward/backward passes can now operate on low bitwidth weights and activations/gradients respectively, DoReFa-Net can use bit convolution kernels to accelerate both training and inference. Moreover, as bit convolutions can be efficiently implemented on CPU, FPGA, ASIC and GPU, DoReFa-Net opens the way to accelerate training of low bitwidth neural network on these hardware. Our experiments on SVHN and ImageNet datasets prove that DoReFa-Net can achieve comparable prediction accuracy as 32-bit counterparts. For example, a DoReFa-Net derived from AlexNet that has 1-bit weights, 2-bit activations, can be trained from scratch using 6-bit gradients to get 46.1% top-1 accuracy on ImageNet validation set. The DoReFa-Net AlexNet model is released publicly.
中文速览
深度神经网络的权重、激活值和梯度通常用32位浮点数表示，这让训练和推断都又慢又费资源。DoReFa-Net 提出了一套完整的低比特量化方案，把权重、激活值和反向传播中的参数梯度（parameter gradients）全部压缩到低位宽整数，其中梯度采用随机量化（stochastic quantization）以避免精度损失，从而让前向和反向计算都能用高效的位运算卷积核（bit convolution kernel）来完成。在 ImageNet 上的实验表明，用1位权重、2位激活值、6位梯度从头训练的 AlexNet 能达到46.1%的 top-1 准确率，与全精度模型相当；在 SVHN 上用1位权重、1位激活值、2位梯度也能达到93%准确率。这项工作首次打通了训练全流程的低比特加速，为在 CPU、FPGA、ASIC 和 GPU 上高效训练低比特网络铺平了道路，对边缘设备部署和降低训练能耗具有重要意义。
原文 arXiv:1606.06160;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.06160v3