Mask-Predict: Parallel Decoding of Conditional Masked Language Models
Marjan Ghazvininejad∗ Omer Levy∗ Yinhan Liu Luke Zettlemoyer Facebook AI Research Seattle, WA ∗Equal contribution, sorted alphabetically.
Abstract
Most machine translation systems generate text autoregressively from left to right. We, instead, use a masked language modeling objective to train a model to predict any subset of the target words, conditioned on both the input text and a partially masked target translation. This approach allows for efficient iterative decoding, where we first predict all of the target words non-autoregressively, and then repeatedly mask out and regenerate the subset of words that the model is least confident about. By applying this strategy for a constant number of iterations, our model improves state-of-the-art performance levels for non-autoregressive and parallel decoding translation models by over 4 BLEU on average. It is also able to reach within about 1 BLEU point of a typical left-to-right transformer model, while decoding significantly faster.111Our code is publicly available at: https://github.com/facebookresearch/Mask-Predict
中文速览
翻译系统通常逐词从左到右生成译文，速度慢且难以并行化。这篇论文提出了一种「条件掩码语言模型」（Conditional Masked Language Model, CMLM），训练时随机遮住目标句子中的部分词，让模型同时根据源文本和已知译文预测所有被遮住的词；解码时采用「掩码-预测」（Mask-Predict）算法：先把整句目标文全部遮住并行生成初稿，再反复把置信度最低的词重新遮住、重新预测，如此迭代固定轮次。实验结果显示，仅需4轮迭代，BLEU分数就超过了此前所有非自回归和并行解码翻译模型，10轮后在WMT'14英德翻译上领先此前最优方法4–5个BLEU分，同时比传统自回归Transformer快约3倍、性能仅下降约1个BLEU点。这项工作证明了在几乎不牺牲翻译质量的前提下大幅提升推理速度是可行的，为机器翻译的高效部署提供了一条切实可行的新路径。
原文 arXiv:1904.09324;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.09324v2