Incorporating BERT into Neural Machine Translation
Jinhua Zhu1,∗, Yingce Xia2, , Lijun Wu3, Di He4, Tao Qin2, Wengang Zhou1, Houqiang Li1, Tie-Yan Liu2 1CAS Key Laboratory of GIPAS, EEIS Department, University of Science and Technology of China; 2Microsoft Research; 3Sun Yat-sen University; 4Key Laboratory of Machine Perception (MOE), School of EECS, Peking University This work is conducted at Microsoft Research Asia. The first two authors contributed equally to this work.
Abstract
The recently proposed BERT (Devlin et al., 2019) has shown great power on a variety of natural language understanding tasks, such as text classification, reading comprehension, etc. However, how to effectively apply BERT to neural machine translation (NMT) lacks enough exploration. While BERT is more commonly used as fine-tuning instead of contextual embedding for downstream language understanding tasks, in NMT, our preliminary exploration of using BERT as contextual embedding is better than using for fine-tuning. This motivates us to think how to better leverage BERT for NMT along this direction. We propose a new algorithm named BERT-fused model, in which we first use BERT to extract representations for an input sequence, and then the representations are fused with each layer of the encoder and decoder of the NMT model through attention mechanisms. We conduct experiments on supervised (including sentence-level and document-level translations), semi-supervised and unsupervised machine translation, and achieve state-of-the-art results on seven benchmark datasets. Our code is available at https://github.com/bert-nmt/bert-nmt.
中文速览
如何把预训练模型BERT真正用好来提升机器翻译质量，是个尚未被充分解决的难题。作者发现直接用BERT初始化翻译模型参数收效甚微，反而将BERT的输出作为上下文特征注入效果更好，由此提出了"BERT融合模型"（BERT-fused model）：先用BERT对源语言句子编码，再通过额外的注意力机制将BERT表示融入翻译模型编码器和解码器的每一层，同时引入"drop-net"技巧防止模型过度依赖某一方的特征。在有监督、半监督和无监督三类机器翻译共14项实验中，该方法在七个标准数据集上均刷新了最优成绩，例如IWSLT'14德英翻译达到36.11 BLEU、WMT'14英德达到30.75 BLEU。这项工作为如何将通用预训练语言模型高效迁移到序列生成任务提供了一套清晰可复用的方案，对整个NLP社区具有重要参考价值。
原文 arXiv:2002.06823;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.06823v1