Depth Growing for Neural Machine Translation
Lijun Wu1,∗, Yiren Wang2,, Yingce Xia3,, Fei Tian3, Fei Gao3, Tao Qin3, Jianhuang Lai1, Tie-Yan Liu3 1School of Data and Computer Science, Sun Yat-sen University; 2 University of Illinois at Urbana-Champaign; 3 Microsoft Research Asia; 1{wulijun3, 3{Yingce.Xia, fetia, feiga, taoqin, The first two authors contributed equally to this work. This work is conducted at Microsoft Research Asia. Corresponding author.
Abstract
While very deep neural networks have shown effectiveness for computer vision and text classification applications, how to increase the network depth of neural machine translation (NMT) models for better translation quality remains a challenging problem. Directly stacking more blocks to the NMT model results in no improvement and even reduces performance. In this work, we propose an effective two-stage approach with three specially designed components to construct deeper NMT models, which results in significant improvements over the strong Transformer baselines on WMT $14$ English $\to$ German and English $\to$ French translation tasks111Our code is available at https://github.com/apeterswu/Depth_Growing_NMT.
中文速览
机器翻译模型越堆越深反而会掉分——直接往 Transformer 上多加几层，性能不升反降，这是该领域长期悬而未决的难题。为此，作者提出了一种两阶段"生长"训练策略：先把一个训练好的 6 层 Transformer 冻住作为底层模块，再在其上新增若干层作为顶层模块，并配合三个专门设计的组件——跨模块残差连接、层级式编码器-解码器注意力（hierarchical encoder-decoder attention）、以及深浅联合解码（deep-shallow decoding）——来让新增的层真正发挥作用。在 WMT 14 英德和英法翻译基准上，该方法分别带来了 1.0 和 0.6 个 BLEU 的提升，不仅超越了直接加层的基线，还以更少的训练代价（仅需额外 8 GPU 天）超过了两模型集成（ensemble）的效果。这项工作为如何在不大幅增加训练成本的前提下突破翻译模型的深度瓶颈提供了一条切实可行的路径。
原文 arXiv:1907.01968;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.01968v1