Exploring Sparsity in Recurrent Neural Networks
Sharan Narang, Erich Elsen, Greg Diamos、Shubho Sengupta Baidu Research Now at Google Brain at Facebook AI Research
Abstract
Recurrent Neural Networks (RNN) are widely used to solve a variety of problems and as the quantity of data and the amount of available compute have increased, so have model sizes. The number of parameters in recent state-of-the-art networks makes them hard to deploy, especially on mobile phones and embedded devices. The challenge is due to both the size of the model and the time it takes to evaluate it. In order to deploy these RNNs efficiently, we propose a technique to reduce the parameters of a network by pruning weights during the initial training of the network. At the end of training, the parameters of the network are sparse while accuracy is still close to the original dense neural network. The network size is reduced by 8 $\times$ and the time required to train the model remains constant. Additionally, we can prune a larger dense network to achieve better than baseline performance while still reducing the total number of parameters significantly. Pruning RNNs reduces the size of the model and can also help achieve significant inference time speed-up using sparse matrix multiply. Benchmarks show that using our technique model size can be reduced by 90% and speed-up is around
中文速览
循环神经网络（RNN）在语音识别等任务中越来越强大，但模型参数动辄上亿，根本没法部署到手机或嵌入式设备上。这篇论文提出了一种"训练中逐步剪枝（gradual pruning during training）"的方法：在正常训练的同时，按照一个单调递增的阈值把绝对值小的权重逐渐置零，训练结束后得到一个高度稀疏的模型，整个过程不需要额外的重新训练，训练耗时也不增加。实验在2100小时英语语音数据上验证了该方法对双向RNN和GRU都有效，最终可将模型参数压缩90%，推理速度提升2到7倍，而且从一个更大的稠密网络出发剪枝，准确率甚至能超过原始基线模型。这项工作为把大规模语音识别模型真正跑在移动端提供了一条简单实用的路径，对工业部署有直接的参考价值。
原文 arXiv:1704.05119;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.05119v2