Recurrent Neural Networks With Limited Numerical Precision
Joachim Ott∗, Zhouhan Lin‡,Ying Zhang‡, Shih-Chii Liu∗, Yoshua Bengio‡† ∗Institute of Neuroinformatics, University of Zurich and ETH Zurich ‡Département d’informatique et de recherche opérationnelle, Université de Montréal †CIFAR Senior Fellow {zhouhan.lin,
Abstract
Recurrent Neural Networks (RNNs) produce state-of-art performance on many machine learning tasks but their demand on resources in terms of memory and computational power are often high. Therefore, there is a great interest in optimizing the computations performed with these models especially when considering development of specialized low-power hardware for deep networks. One way of reducing the computational needs is to limit the numerical precision of the network weights and biases. This has led to different proposed rounding methods which have been applied so far to only Convolutional Neural Networks and Fully-Connected Networks. This paper addresses the question of how to best reduce weight precision during training in the case of RNNs. We present results from the use of different stochastic and deterministic reduced precision training methods applied to three major RNN types which are then tested on several datasets. The results show that the weight binarization methods do not work with the RNNs. However, the stochastic and deterministic ternarization, and pow2-ternarization methods gave rise to low-precision RNNs that produce similar and even higher accuracy on certain datase
中文速览
循环神经网络（RNN）在语音识别、机器翻译等任务上表现出色，但其对内存和算力的需求极高，限制了在低功耗硬件上的部署。研究者尝试通过将网络权重压缩到极低精度（如只用-1和1表示的二值化，或-1、0、1的三值化）来降低计算量，但此前这类方法仅在卷积网络和全连接网络上验证过，能否用于RNN训练仍是未解之题。本文系统评估了随机与确定性二值化、三值化、2次幂三值化以及新提出的指数量化四种方法，分别应用于Vanilla RNN、LSTM和GRU三种主流架构，并在字符级语言建模和语音识别等多个数据集上测试。结果发现二值化方法会导致RNN隐状态爆炸或消失而完全失效，而三值化及相关方法不仅保住了精度，在部分数据集上甚至超过了全精度基线，这为在专用低功耗芯片上高效运行RNN开辟了可行路径。
原文 arXiv:1608.06902;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1608.06902v2