Synthesizer: Rethinking Self-Attention for Transformer Models
Yi Tay Dara Bahri Donald Metzler Da-Cheng Juan Zhe Zhao Che Zheng
Abstract
The dot product self-attention is known to be central and indispensable to state-of-the-art Transformer models. But is it really required? This paper investigates the true importance and contribution of the dot product-based self-attention mechanism on the performance of Transformer models. Via extensive experiments, we find that (1) random alignment matrices surprisingly perform quite competitively and (2) learning attention weights from token-token (query-key) interactions is useful but not that important after all. To this end, we propose Synthesizer, a model that learns synthetic attention weights without token-token interactions. In our experiments, we first show that simple Synthesizers achieve highly competitive performance when compared against vanilla Transformer models across a range of tasks, including machine translation, language modeling, text generation and GLUE/SuperGLUE benchmarks. When composed with dot product attention, we find that Synthesizers consistently outperform Transformers. Moreover, we conduct additional comparisons of Synthesizers against Dynamic Convolutions, showing that simple Random Synthesizer is not only $60\%$ faster but also improves perplexit
中文速览
Transformer模型的成功通常被归功于点积自注意力（dot product self-attention）机制，但这篇论文质疑它是否真的不可或缺。研究者提出了一种名为Synthesizer的新模型，用"合成注意力"替代传统的查询-键-值点积运算：有的变体只依赖单个词元的信息生成注意力矩阵，有的变体甚至直接使用随机初始化的权重矩阵而完全不看输入内容。实验结果出人意料——随机权重矩阵的Synthesizer在机器翻译上达到了与标准Transformer几乎相同的BLEU分数，在语言建模、文本生成及GLUE/SuperGLUE等多项基准上也表现相当，与Dynamic Convolution相比速度快60%且困惑度降低3.5%。这项研究表明，长期被视为Transformer核心的词元间点积交互并非性能的唯一来源，为重新理解自注意力的真实作用、以及设计更高效的注意力机制提供了重要启示。
原文 arXiv:2005.00743;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.00743v3