Online and Linear-Time Attention by Enforcing Monotonic Alignments
Colin Raffel Minh-Thang Luong Peter J. Liu Ron J. Weiss Douglas Eck
Abstract
Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time. We validate our approach on sentence summarization, machine translation, and online speech recognition problems and achieve results competitive with existing sequence-to-sequence models.
中文速览
序列到序列模型中的软注意力（soft attention）每生成一个输出都要扫描整段输入，导致计算复杂度是输入与输出长度的乘积，也无法用于实时（online）场景。针对语音识别、机器翻译等输入输出天然单调对齐的任务，本文提出了一种"单调注意力"机制：在解码时从上次停下的位置继续向右逐步检查输入，一旦决定"停止"便直接读取当前位置作为上下文，从而把推理复杂度降到线性并支持在线解码；训练时则通过计算该随机过程的期望输出，将原本不可微的硬对齐转化为可用标准反向传播优化的平滑目标。在句子摘要、机器翻译和在线语音识别三项任务上，该方法的性能与传统软注意力模型相当，却首次实现了端到端可训练的线性时间在线注意力，对于延迟敏感的实时语音等应用具有重要意义。
原文 arXiv:1704.00784;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.00784v2