Insertion-based Decoding with automatically Inferred Generation Order
Jiatao Gu†, Qi Liu⋄, and Kyunghyun Cho‡† †Facebook AI Research ⋄University of Oxford ‡New York University, CIFAR Azrieli Global Scholar †{jgu, This work was completed while the author worked as an AI resident at Facebook AI Research.
Abstract
Conventional neural autoregressive decoding commonly assumes a fixed left-to-right generation order, which may be sub-optimal. In this work, we propose a novel decoding algorithm – InDIGO – which supports flexible sequence generation in arbitrary orders through insertion operations. We extend Transformer, a state-of-the-art sequence generation model, to efficiently implement the proposed approach, enabling it to be trained with either a pre-defined generation order or adaptive orders obtained from beam-search. Experiments on four real-world tasks, including word order recovery, machine translation, image caption and code generation, demonstrate that our algorithm can generate sequences following arbitrary orders, while achieving competitive or even better performance compared to the conventional left-to-right generation. The generated sequences show that InDIGO adopts adaptive generation orders based on input information.
中文速览
自然语言生成模型通常从左到右逐字生成句子，但这种固定顺序未必是最优的——有时先确定关键词再填充周边词效果更好。本文提出了一种名为 InDIGO（Insertion-based Decoding with Inferred Generation Order）的新解码算法，把生成顺序建模为隐变量，通过每步同时预测"插入哪个词"和"插到哪个位置"来自动推断最优生成顺序，并在 Transformer 中用相对位置注意力高效实现，支持用预定义顺序或束搜索得到的自适应顺序进行训练。在词序恢复、机器翻译、图像描述和代码生成四项任务上的实验表明，InDIGO 能按任意顺序生成序列，性能与传统从左到右生成方式持平甚至更优，且案例分析显示模型确实能根据输入内容自适应地选择生成顺序。这项工作说明打破固定从左到右的生成假设是可行且有益的，为序列生成提供了更灵活的范式。
原文 arXiv:1902.01370;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.01370v3