Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation
Biao Zhang1 Philip Williams1 Ivan Titov1,2 Rico Sennrich3,1 1School of Informatics, University of Edinburgh 2ILLC, University of Amsterdam 3Department of Computational Linguistics, University of Zurich
Abstract
Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. In this paper, we explore ways to improve them. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics, and overcome this bottleneck via language-specific components and deepening NMT architectures. We identify the off-target translation issue (i.e. translating into a wrong target language) as the major source of the inferior zero-shot performance, and propose random online backtranslation to enforce the translation of unseen training language pairs. Experiments on OPUS-100 (a novel multilingual dataset with 100 languages) show that our approach substantially narrows the performance gap with bilingual models in both one-to-many and many-to-many settings, and improves zero-shot performance by $\sim$ 10 BLEU, approaching conventional pivot-based methods.111We release our code at https://github.com/bzhangGo/zero. We release the OPUS-100 dataset at https://github.com/EdinburghNLP/opus-100-corpus.
中文速览
大规模多语言神经机器翻译（multilingual NMT）虽然理论上用一个模型就能搞定上百种语言互译，但实际效果往往不如单独训练的双语模型，尤其是在训练时从未见过的语言对（zero-shot）上翻译质量极差。针对这两个痛点，研究者提出了两类改进方案：一是通过引入语言感知的层归一化（language-aware layer normalization）和线性变换、以及加深Transformer网络结构来增强模型容量，让不同语言不再被强行挤进同一表示空间；二是发现zero-shot效果差的根源在于模型经常把目标语言搞错（off-target translation），为此提出了随机在线回译算法（random online backtranslation，ROBt），每次训练时随机挑一种中间语言做回译，生成伪平行数据来逼迫模型学会它从未直接见过的翻译方向。在覆盖100种语言的新数据集OPUS-100上进行实验，结果显示容量增强方案大幅缩小了与双语模型的性能差距，而ROBt将off-target翻译比例降低约50%、zero-shot BLEU分数提升约10分，接近传统的以英语为枢纽语言的pivot方法。这项工作首次在百种语言规模上系统验证了上述方法的有效性，并公开了数据集和代码，为后续多语言翻译研究奠定了重要基础。
原文 arXiv:2004.11867;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.11867v1