Semi-Supervised Learning for Neural Machine Translation
Yong Cheng#, Wei Xu#, Zhongjun He+, Wei He+, Hua Wu+, Maosong Sun† and Yang Liu† #Institute for Interdisciplinary Information Sciences, Tsinghua University, Beijing, China †State Key Laboratory of Intelligent Technology and Systems Tsinghua National Laboratory for Information Science and Technology Department of Computer Science and Technology, Tsinghua University, Beijing, China +Baidu Inc., Beijing, China Yang Liu is the corresponding author.
Abstract
While end-to-end neural machine translation (NMT) has made remarkable progress recently, NMT systems only rely on parallel corpora for parameter estimation. Since parallel corpora are usually limited in quantity, quality, and coverage, especially for low-resource languages, it is appealing to exploit monolingual corpora to improve NMT. We propose a semi-supervised approach for training NMT models on the concatenation of labeled (parallel corpora) and unlabeled (monolingual corpora) data. The central idea is to reconstruct the monolingual corpora using an autoencoder, in which the source-to-target and target-to-source translation models serve as the encoder and decoder, respectively. Our approach can not only exploit the monolingual corpora of the target language, but also of the source language. Experiments on the Chinese-English dataset show that our approach achieves significant improvements over state-of-the-art SMT and NMT systems.
中文速览
神经机器翻译（NMT）系统高度依赖双语平行语料库来训练模型，而平行语料普遍稀缺、覆盖域窄，严重制约了翻译质量。研究者提出了一种半监督训练方法：在有标注的平行语料之外，同时利用无标注的单语语料——核心思路是构建一个自编码器（autoencoder），用"目标语→源语"翻译模型将单语句子编码成隐含的另一语言句子，再用"源语→目标语"翻译模型将其解码还原，以重建原始单语句子，从而让两个方向的翻译模型在联合训练中相互促进。由于推断空间是指数级的，他们用top-k候选列表（k=10）近似完整搜索空间来保证训练效率。在中英互译的NIST评测集上，该方法在两个翻译方向均显著优于当时最强的统计机器翻译系统Moses和注意力NMT系统RNNSearch，表明同时利用源语和目标语单语语料是提升低资源场景下NMT质量的有效途径。
原文 arXiv:1606.04596;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.04596v3