A Cheap Linear Attention Mechanism with Fast Lookups and Fixed-Size Representations
Alexandre de Brébisson MILA, University of Montréal、Pascal Vincent MILA, University of Montréal and CIFAR
Abstract
The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not encode the sequence into a fixed-size representation but instead requires to memorize all the hidden states. These two limitations restrict the use of the softmax attention mechanism to relatively small-scale applications with short sequences and few lookups per sequence. In this work we introduce a family of linear attention mechanisms designed to overcome the two limitations listed above. We show that removing the softmax non-linearity from the traditional attention formulation yields constant-time attention lookups and fixed-size representations of the attended sequences. These properties make these linear attention mechanisms particularly suitable for large-scale applications with extreme query loads, real-time requirements and memory constraints. Early experiments on a question answering task show that these linear mechanisms yield significantly better accuracy resu
中文速览
传统的软性注意力机制（softmax attention）虽然在循环神经网络中效果出色，但每次查询都要扫描一遍整个文档序列，计算量和内存占用都随序列长度线性增长，无法满足大规模实时场景的需求。为此，本文提出了一族"线性注意力机制"——核心思路是去掉 softmax 非线性，把文档的所有隐状态压缩成一个固定大小的 k×k 协方差矩阵 C，使得后续每次查询只需 O(k²) 的常数时间，而无需再扫描原始序列；在此基础上还引入了门控版本，通过可学习的门控函数控制信息写入 C 的方式，以弥补部分精度损失。在 CNN 问答数据集上的实验表明，线性注意力机制的准确率明显优于不加注意力的基线，虽然仍低于 softmax 注意力，但门控扩展能缩小这一差距。这项工作为需要处理海量并发查询、有严格实时要求和内存限制的大规模信息检索场景提供了一条可行的轻量化注意力路径。
原文 arXiv:1609.05866;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1609.05866v1