Non-Autoregressive Machine Translation with Auxiliary Regularization
1Yiren Wang, 2Fei Tian, 3Di He, 2Tao Qin, 1ChengXiang Zhai, 2Tie-Yan Liu 1University of Illinois at Urbana-Champaign, Urbana, IL, USA 2Microsoft Research, Beijing, China 3Key Laboratory of Machine Perception, MOE, School of EECS, Peking University, Beijing, China 2{fetia, taoqin, The work was done when the author was an intern at Microsoft Research Asia.
Abstract
As a new neural machine translation approach, Non-Autoregressive machine Translation (NAT) has attracted attention recently due to its high efficiency in inference. However, the high efficiency has come at the cost of not capturing the sequential dependency on the target side of translation, which causes NAT to suffer from two kinds of translation errors: 1) repeated translations (due to indistinguishable adjacent decoder hidden states), and 2) incomplete translations (due to incomplete transfer of source side information via the decoder hidden states). In this paper, we propose to address these two problems by improving the quality of decoder hidden representations via two auxiliary regularization terms in the training process of an NAT model. First, to make the hidden states more distinguishable, we regularize the similarity between consecutive hidden states based on the corresponding target tokens. Second, to force the hidden states to contain all the information in the source sentence, we leverage the dual nature of translation tasks (e.g., English to German and German to English) and minimize a backward reconstruction error to ensure that the hidden states of the NAT decoder a
中文速览
非自回归翻译（Non-Autoregressive Translation, NAT）模型因为能并行生成所有词、推理速度极快而备受关注，但它放弃了目标语言端的序列依赖，导致解码器隐藏状态质量低下，频繁出现"重复翻译"（相邻位置反复生成同一个词）和"漏译"（源端部分信息未被传递）两类错误。针对这两个问题，作者在训练阶段引入两项辅助正则化：一是"相似度正则化"，让相邻隐藏状态之间的相似程度与对应目标词嵌入之间的相似程度保持一致，从而使相邻隐藏状态更易区分；二是"重构正则化"，利用翻译任务的对偶性，在解码器隐藏状态之上附加一个自回归反向翻译模型，要求这些隐藏状态能够还原源端句子，从而迫使它们保留完整的源端信息。两项正则化仅在训练时生效、推理时完全不引入额外开销，在WMT14英德和德英基准上分别取得24.61和28.90的BLEU分数，刷新了NAT模型的最优记录，证明了在几乎不损失推理效率的前提下大幅提升翻译质量的可行性。
原文 arXiv:1902.10245;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.10245v1