Pragmatic Neural Language Modelling in Machine Translation
Paul Baltescu University of Oxford、Phil Blunsom University of Oxford Google DeepMind
Abstract
This paper presents an in-depth investigation on integrating neural language models in translation systems. Scaling neural language models is a difficult task, but crucial for real-world applications. This paper evaluates the impact on end-to-end MT quality of both new and existing scaling techniques. We show when explicitly normalising neural models is necessary and what optimisation tricks one should use in such scenarios. We also focus on scalable training algorithms and investigate noise contrastive estimation and diagonal contexts as sources for further speed improvements. We explore the trade-offs between neural models and back-off n-gram models and find that neural models make strong candidates for natural language applications in memory constrained environments, yet still lag behind traditional models in raw translation quality. We conclude with a set of recommendations one should follow to build a scalable neural language model for MT.
中文速览
把神经语言模型（neural language model）用进机器翻译（machine translation）解码器时，softmax层的计算开销极大，严重制约了实用化进程，这篇论文专门研究如何在保证翻译质量的前提下让神经语言模型跑得更快、占内存更少。作者系统评估了多种加速方案，包括基于Brown聚类的词类分解、树形因子化、以及噪声对比估计（noise contrastive estimation），并重点探索了将噪声对比估计与词类分解相结合的新训练策略，同时考察了对角上下文矩阵等优化技巧。实验结果表明，在纯BLEU分数上神经模型仍然落后于传统Kneser-Ney n-gram模型，但一旦把内存限制纳入考量，神经模型就明显更具优势，非常适合手机等内存受限的场景。这项工作给出了一套构建可扩展神经语言模型的实用建议，对于在真实系统中部署神经语言模型具有直接的指导价值。
原文 arXiv:1412.7119;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1412.7119v3