Scaling Recurrent Neural Network Language Models
Abstract
This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model size, training-set size, computational costs and memory. Our analysis shows that despite being more costly to train, RNNLMs obtain much lower perplexities on standard benchmarks than $n$ -gram models. We train the largest known RNNs and present relative word error rates gains of 18% on an ASR task. We also present the new lowest perplexities on the recently released billion word language modelling benchmark, 1 BLEU point gain on machine translation and a 17% relative hit rate gain in word prediction.
中文速览
循环神经网络语言模型（RNNLM）长期因计算量太大而难以扩展，研究者们想搞清楚它到底能不能随着模型规模和数据量的增大持续变强。本文通过把大规模 RNNLM 搬上 GPU 训练，系统分析了模型大小、训练数据量、算力和内存四个维度的扩展规律，并训练出了当时已知最大的 RNN（隐层维度达 8192）。实验结果显示，RNNLM 的困惑度（perplexity）比传统 5-gram 模型低 40% 以上，同时参数量还不到后者的一半；在语音识别任务上词错误率相对降低 18%，机器翻译 BLEU 分提升 1 分，手机输入法预测命中率提升 17%。这项工作证明了 RNNLM 在实际应用场景中全面超越 n-gram 的潜力，也为后续更大规模神经网络语言模型的发展奠定了基础。
原文 arXiv:1502.00512;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1502.00512v1