Deterministic Non-Autoregressive Neural Sequence Modeling by Iterative Refinement
Jason Lee New York University、Elman Mansimov††New York University、Kyunghyun Cho New York University CIFAR Azrieli Global Scholar Equal Contribution
Abstract
We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine translation (En $\leftrightarrow$ De and En $\leftrightarrow$ Ro) and image caption generation, and observe that it significantly speeds up decoding while maintaining the generation quality comparable to the autoregressive counterpart.
中文速览
用迭代精修（iterative refinement）来加速序列生成是这项工作的核心思路：传统自回归模型逐词生成文本，解码慢且难以并行，而这篇文章提出了一种非自回归神经序列模型，把生成过程看作对粗糙草稿的反复打磨——先一次性生成全部目标词，再通过共享参数的网络多轮迭代修正。模型同时借鉴了隐变量模型和去噪自编码器（denoising autoencoder）两种思路，训练时混合两类损失函数，并配合知识蒸馏（knowledge distillation）来弥补非自回归建模的质量损失。在机器翻译（英德、英罗）和图片描述生成任务上，该方法的解码速度比自回归的Transformer快2–5倍，同时保留了90%–95%左右的翻译质量。这项工作的重要性在于，它提供了一条通用的、不依赖额外对齐工具或重排序机制的非自回归序列生成框架，为在实际部署中降低推理延迟指出了可行路径。
原文 arXiv:1802.06901;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.06901v3