Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
Yuhang Li ††{\dagger} , Xin Dong§§\S111 , Wei Wang ††{\dagger} ††{\dagger}National University of Singapore, §§\SHarvard University Equal Contribution. Y. L. completed this work during his internship at NUS.
Abstract
We propose Additive Powers-of-Two (APoT) quantization, an efficient non-uniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks. By constraining all quantization levels as the sum of Powers-of-Two terms, APoT quantization enjoys high computational efficiency and a good match with the distribution of weights. A simple reparameterization of the clipping function is applied to generate a better-defined gradient for learning the clipping threshold. Moreover, weight normalization is presented to refine the distribution of weights to make the training more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models, demonstrating the effectiveness of our proposed APoT quantization. For example, our 4-bit quantized ResNet-50 on ImageNet achieves 76.6% top-1 accuracy without bells and whistles; meanwhile, our model reduces 22% computational cost compared with the uniformly quantized counterpart. 111Code is available at https://github.com/yhhhli/APoT_Quantization.
中文速览
神经网络权重和激活值的分布是钟形长尾的，现有量化方案要么用均匀量化（计算友好但精度损失大），要么用2的幂次量化（分辨率随比特位增加时会过度集中在零附近）。研究者提出了"加性2的幂次量化"（Additive Powers-of-Two, APoT），把每个量化级别设计为多个2的幂次项之和，既保留了移位运算代替乘法的硬件加速优势，又让量化级别在整个数值范围内分布更合理、更贴合权重的真实分布；同时还提出了重参数化裁剪函数（RCF）和权重归一化，让裁剪阈值在训练中得到更准确的梯度并更稳定地优化。在ImageNet上，4比特APoT量化的ResNet-50达到76.6%的top-1精度，与全精度模型基本持平，同时比均匀量化节省22%的计算量，证明这一方案在精度和效率上都优于当时的主流方法。
原文 arXiv:1909.13144;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.13144v2