Imputer: Sequence Modelling via Imputation and Dynamic Programming
William Chan Chitwan Saharia Geoffrey Hinton Mohammad Norouzi Navdeep Jaitly
Abstract
This paper presents the Imputer, a neural sequence model that generates output sequences iteratively via imputations. The Imputer is an iterative generative model, requiring only a constant number of generation steps independent of the number of input or output tokens. The Imputer can be trained to approximately marginalize over all possible alignments between the input and output sequences, and all possible generation orders. We present a tractable dynamic programming training algorithm, which yields a lower bound on the log marginal likelihood. When applied to end-to-end speech recognition, the Imputer outperforms prior non-autoregressive models and achieves competitive results to autoregressive models. On LibriSpeech test-other, the Imputer achieves 11.1 WER, outperforming CTC at 13.0 WER and seq2seq at 12.5 WER.
中文速览
语音识别模型通常面临一个两难困境：自回归模型（autoregressive model）效果好但生成速度慢，需要逐个输出词元；非自回归模型（non-autoregressive model）速度快却因词元间相互独立假设而精度偏低。本文提出 Imputer，一种基于"填空"思想的迭代生成模型：它将输入（音频）与输出（文本）的对齐序列叠加在一起，每轮推理从部分填好的序列出发，并行预测剩余位置的词元，经过固定的常数轮迭代即可完成完整序列的生成，与输出长度无关。训练上，作者设计了一套可处理的动态规划算法，对所有可能对齐方式和生成顺序进行近似边际化，得到对数边际似然的下界，从而兼顾对齐多样性与生成依赖建模。在 LibriSpeech 端到端语音识别基准上，Imputer 在 test-other 集上取得 11.1% 的词错误率（WER），不仅超越 CTC 的 13.0% 和 seq2seq 的 12.5%，还将推理步数压缩为常数级，为在长序列场景下同时追求速度与精度提供了新的可行路径。
原文 arXiv:2002.08926;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.08926v2