Deep Learning with Limited Numerical Precision
Suyog Gupta Ankur Agrawal Kailash Gopalakrishnan IBM T. J. Watson Research Center, Yorktown Heights, NY 10598 Pritish Narayanan IBM Almaden Research Center, San Jose, CA 95120
Abstract
Training of large-scale deep neural networks is often constrained by the available computational resources. We study the effect of limited precision data representation and computation on neural network training. Within the context of low-precision fixed-point computations, we observe the rounding scheme to play a crucial role in determining the network’s behavior during training. Our results show that deep networks can be trained using only $16$ -bit wide fixed-point number representation when using stochastic rounding, and incur little to no degradation in the classification accuracy. We also demonstrate an energy-efficient hardware accelerator that implements low-precision fixed-point arithmetic with stochastic rounding.
中文速览
深度神经网络训练通常依赖32位浮点数运算，但这对硬件资源消耗巨大。研究者探索了用16位定点数（fixed-point）替代32位浮点数来训练深度网络的可行性，发现关键在于采用随机舍入（stochastic rounding）——即按照数值与最近可表示值的距离概率性地决定舍入方向，而非传统的"四舍五入"。实验在MNIST和CIFAR10图像分类任务上表明，配合随机舍入的16位定点训练几乎不损失分类精度，与32位浮点基准持平，而采用普通舍入方式则会导致训练失败或精度明显下滑。研究者还在FPGA上实现了支持随机舍入的低精度定点硬件加速器原型，验证了这一方案在实际硬件中能同时获得高吞吐量与低功耗，对未来专用AI训练芯片的设计具有重要参考价值。
原文 arXiv:1502.02551;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1502.02551v1