Levenshtein Transformer
Jiatao Gu†, Changhan Wang†, and Jake Zhao (Junbo)‡⋄ †Facebook AI Research ‡New York University ⋄Tigerobo Inc. †{jgu,
Abstract
Modern neural sequence generation models are built to either generate tokens step-by-step from scratch or (iteratively) modify a sequence of tokens bounded by a fixed length. In this work, we develop Levenshtein Transformer, a new partially autoregressive model devised for more flexible and amenable sequence generation. Unlike previous approaches, the basic operations of our model are insertion and deletion. The combination of them facilitates not only generation but also sequence refinement allowing dynamic length changes. We also propose a set of new training techniques dedicated at them, effectively exploiting one as the other’s learning signal thanks to their complementary nature. Experiments applying the proposed model achieve comparable or even better performance with much-improved efficiency on both generation (e.g. machine translation, text summarization) and refinement tasks (e.g. automatic post-editing). We further confirm the flexibility of our model by showing a Levenshtein Transformer trained by machine translation can straightforwardly be used for automatic post-editing. 111Codes for reproducing this paper are released in https://github.com/pytorch/fairseq/tree/master
中文速览
传统神经序列生成模型要么逐步自回归地生成词元，要么在固定长度内迭代修改序列，缺乏像人类写作那样随意增删修改的灵活性。为此，研究者提出了 Levenshtein Transformer（LevT），以"插入"和"删除"这两个基本操作替代原有的解码机制，并设计了一套"双策略模仿学习"算法，让两个互补的策略互相提供训练信号，从而高效学习何时插入、何时删除。在机器翻译、文本摘要等生成任务上，LevT 达到与强基线持平甚至更优的翻译质量，同时解码速度最高提升 5 倍；更难得的是，同一个用机器翻译训练好的模型无需任何修改即可直接用于自动译后编辑这一序列精化任务。这项工作的意义在于首次将序列生成与序列精化统一在同一框架下，打破了两类任务因模型归纳偏置不同而必须分开建模的惯例。
原文 arXiv:1905.11006;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.11006v2