Learning Deep Transformer Models for Machine Translation
Qiang Wang1, Bei Li1, Tong Xiao1,2, Jingbo Zhu1,2, Changliang Li3, Derek F. Wong4, Lidia S. Chao4 1NLP Lab, Northeastern University, Shenyang, China 2NiuTrans Co., Ltd., Shenyang, China 3Kingsoft AI Lab, Beijing, China 4NLP2CT Lab, University of Macau, Macau, China Corresponding author.
Abstract
Transformer is the state-of-the-art model in recent machine translation evaluations. Two strands of research are promising to improve models of this kind: the first uses wide networks (a.k.a. Transformer-Big) and has been the de facto standard for the development of the Transformer system, and the other uses deeper language representation but faces the difficulty arising from learning deep networks. Here, we continue the line of research on the latter. We claim that a truly deep Transformer model can surpass the Transformer-Big counterpart by 1) proper use of layer normalization and 2) a novel way of passing the combination of previous layers to the next. On WMT’16 English-German, NIST OpenMT’12 Chinese-English and larger WMT’18 Chinese-English tasks, our deep system (30/25-layer encoder) outperforms the shallow Transformer-Big/Base baseline (6-layer encoder) by 0.4 $\sim$ 2.4 BLEU points. As another bonus, the deep model is 1.6X smaller in size and 3X faster in training than Transformer-Big111The source code is available at https://github.com/wangqiangneu/dlcl.
中文速览
训练更深的神经机器翻译（NMT）编码器一直面临梯度消失等优化难题，简单叠加层数往往导致训练失败、性能下滑。研究者提出了两项关键改进：一是将层归一化（Layer Normalization）从残差相加之后移到子层输入之前（Pre-Norm），使梯度能直接从顶层反传到底层，从根本上缓解深层网络的训练困难；二是受数值分析中线性多步法启发，设计了"层动态线性组合"（Dynamic Linear Combination of Layers，DLCL）机制，让每一层的输入都以可学习权重融合此前所有层的输出，避免深层模型"遗忘"低层特征。凭借这两项改进，研究者成功训练出30层编码器，在WMT英德、NIST中英及WMT'18中英三项翻译任务上，BLEU值比标准6层Transformer-Base提升1.3–2.4分，甚至超过参数量大得多的Transformer-Big达0.4–0.6分，同时模型参数量缩小1.6倍、训练速度提升3倍。这项工作表明，深度而非宽度才是提升Transformer翻译质量的更高效路径，为后续深层序列建模研究提供了重要参考。
原文 arXiv:1906.01787;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.01787v1