Retrieving Sequential Information for Non-Autoregressive Neural Machine Translation
Chenze Shao123, Yang Feng12⋆, Jinchao Zhang3, Fandong Meng3, Xilin Chen12 and Jie Zhou3 1 University of Chinese Academy of Sciences 2 Key Laboratory of Intelligent Information Processing Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS) 3 Pattern Recognition Center, WeChat AI, Tencent Inc, China {shaochenze18z, fengyang, {dayerzhang, fandongmeng,
Abstract
Non-Autoregressive Transformer (NAT) aims to accelerate the Transformer model through discarding the autoregressive mechanism and generating target words independently, which fails to exploit the target sequential information. Over-translation and under-translation errors often occur for the above reason, especially in the long sentence translation scenario. In this paper, we propose two approaches to retrieve the target sequential information for NAT to enhance its translation ability while preserving the fast-decoding property. Firstly, we propose a sequence-level training method based on a novel reinforcement algorithm for NAT (Reinforce-NAT) to reduce the variance and stabilize the training procedure. Secondly, we propose an innovative Transformer decoder named FS-decoder to fuse the target sequential information into the top layer of the decoder. Experimental results on three translation tasks show that the Reinforce-NAT surpasses the baseline NAT system by a significant margin on BLEU without decelerating the decoding speed and the FS-decoder achieves comparable translation performance to the autoregressive Transformer with considerable speedup. †† Joint work with Pattern Rec
中文速览
非自回归Transformer（Non-Autoregressive Transformer, NAT）能并行生成所有目标词、大幅提升翻译速度，但正因为抛弃了逐词生成的顺序依赖，它容易出现重复、漏译、多译等错误，译文质量与传统自回归模型相比差距明显。为此，作者提出两个方法把目标序列的顺序信息"找回来"：一是设计了一种专为NAT定制的强化学习训练算法Reinforce-NAT，用BLEU等句级指标替代逐词交叉熵损失，并通过"遍历高概率词＋采样低概率词"的方式大幅降低梯度估计的方差、稳定训练；二是提出FS-decoder，让解码器的底层仍以非自回归方式并行运行，只在最顶层引入自回归融合，把序列顺序信息注入最终预测。在IWSLT16和WMT14/16等三个翻译任务上的实验表明，Reinforce-NAT在不损失解码速度的前提下显著提升了NAT的BLEU值，FS-decoder则在保持可观加速比的同时达到了与完整自回归Transformer相当的翻译质量。这项工作为如何在不牺牲速度优势的情况下弥补NAT的质量短板提供了切实可行的方案。
原文 arXiv:1906.09444;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.09444v1