Pay Attention to MLPs
Hanxiao Liu, Zihang Dai, David R. So, Quoc V. Le Google Research, Brain Team
Abstract
Transformers [1] have become one of the most important architectural innovations in deep learning and have enabled many breakthroughs over the past few years. Here we propose a simple network architecture, gMLP, based on MLPs with gating, and show that it can perform as well as Transformers in key language and vision applications. Our comparisons show that self-attention is not critical for Vision Transformers, as gMLP can achieve the same accuracy. For BERT, our model achieves parity with Transformers on pretraining perplexity and is better on some downstream NLP tasks. On finetuning tasks where gMLP performs worse, making the gMLP model substantially larger can close the gap with Transformers. In general, our experiments show that gMLP can scale as well as Transformers over increased data and compute.
中文速览
自注意力机制（self-attention）真的是Transformer成功的核心吗？谷歌Brain团队对此提出质疑，并设计了一种名为gMLP的新架构：它抛弃自注意力，仅用带门控（gating）的多层感知机来捕捉序列中不同位置之间的交互，核心部件是一个"空间门控单元"（Spatial Gating Unit，SGU），以静态矩阵投影替代动态生成的注意力权重。在图像分类任务上，gMLP与Vision Transformer（ViT）精度相当，且参数量比MLP-Mixer少66%的同时准确率反高出3%；在BERT语言预训练任务上，gMLP的困惑度（perplexity）与Transformer持平，在多项下游NLP任务上甚至超过原版BERTlarge。对于少数仍存在差距的任务，只需加入极少量自注意力（单头、维度不超过128）便可弥补，或直接扩大模型规模即可追上。这项研究揭示了自注意力并非深度学习模型扩展的必要条件，为设计更简洁、高效的大规模模型提供了新思路。
原文 arXiv:2105.08050;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.08050v2