Massive Exploration of Neural Machine Translation Architectures
Denny Britz , Anna Goldie, Minh-Thang Luong, Quoc Le Google Brain Both authors contributed equally to this work.Work done as a member of the Google Brain Residency program (g.co/brainresidency).
Abstract
Neural Machine Translation (NMT) has shown remarkable progress over the past few years with production systems now being deployed to end-users. One major drawback of current architectures is that they are expensive to train, typically requiring days to weeks of GPU time to converge. This makes exhaustive hyperparameter search, as is commonly done with other neural network architectures, prohibitively expensive. In this work, we present the first large-scale analysis of NMT architecture hyperparameters. We report empirical results and variance numbers for several hundred experimental runs, corresponding to over 250,000 GPU hours on the standard WMT English to German translation task. Our experiments lead to novel insights and practical advice for building and extending NMT architectures. As part of this contribution, we release an open-source NMT framework111https://github.com/google/seq2seq/ that enables researchers to easily experiment with novel techniques and reproduce state of the art results.
中文速览
神经机器翻译（Neural Machine Translation, NMT）模型效果虽好，但训练一次动辄耗费数天乃至数周的GPU算力，导致研究者几乎无法像其他深度学习任务那样对超参数进行系统性搜索。为此，研究团队投入超过25万GPU小时，在英德翻译标准测试集上对数百组NMT架构超参数配置展开了迄今最大规模的对比实验。实验揭示了一系列实用结论：LSTM单元优于GRU，深层编码器比解码器更难优化，稠密残差连接（dense residual connections）比普通残差连接收敛更快且效果更好，精心调校的束搜索（beam search）对最终翻译质量至关重要，而嵌入维度在一定范围内对BLEU分数的影响远小于预期。这项工作不仅为研究者选择基础架构提供了直接可用的参考，还量化了随机初始化带来的结果波动，帮助区分真实提升与随机噪声，同时配套开源了可复现的seq2seq训练框架，对后续NMT研究具有重要的基准价值。
原文 arXiv:1703.03906;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.03906v2