Augmenting Self-attention with Persistent Memory
Sainbayar Sukhbaatar, Edouard Grave, Guillaume Lample, Herve Jegou, Armand Joulin Facebook AI Research
Abstract
Transformer networks have lead to important progress in language modeling and machine translation. These models include two consecutive modules, a feed-forward layer and a self-attention layer. The latter allows the network to capture long term dependencies and are often regarded as the key ingredient in the success of Transformers. Building upon this intuition, we propose a new model that solely consists of attention layers. More precisely, we augment the self-attention layers with persistent memory vectors that play a similar role as the feed-forward layer. Thanks to these vectors, we can remove the feed-forward layer without degrading the performance of a transformer. Our evaluation shows the benefits brought by our model on standard character and word level language modeling benchmarks.
中文速览
自注意力（self-attention）机制常被视为Transformer成功的核心，但事实上占据大量参数的前馈层（feed-forward layer）同样不可或缺。本文提出一种"全注意力层"（all-attention layer），通过在自注意力中引入一组不依赖输入上下文的持久记忆向量（persistent memory vectors）来取代前馈层，从而将两个子层合并为单一的注意力层，大幅简化网络结构。在标准字符级和词级语言建模基准测试上，该模型取得了与原始Transformer相当甚至更优的性能，且参数量保持不变。这项工作揭示了前馈层本质上可以被视为一种特殊的注意力机制，为重新理解和精简Transformer架构提供了新视角。
原文 arXiv:1907.01470;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.01470v1