Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet
Luke Melas-Kyriazi Oxford University
Abstract
The strong performance of vision transformers on image classification and other vision tasks is often attributed to the design of their multi-head attention layers. However, the extent to which attention is responsible for this strong performance remains unclear. In this short report, we ask: is the attention layer even necessary? Specifically, we replace the attention layer in a vision transformer with a feed-forward layer applied over the patch dimension. The resulting architecture is simply a series of feed-forward layers applied over the patch and feature dimensions in an alternating fashion. In experiments on ImageNet, this architecture performs surprisingly well: a ViT/DeiT-base-sized model obtains 74.9% top-1 accuracy, compared to 77.9% and 79.9% for ViT and DeiT respectively. These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought. We hope these results prompt the community to spend more time trying to understand why our current models are as effective as they are.111Code is available at https://github.com/lukemelas/do-you-even-need-attention
中文速览
视觉Transformer（Vision Transformer, ViT）在图像分类上表现出色，学界普遍认为这要归功于其多头注意力（multi-head attention）机制，但这一假设从未被直接验证过。研究者做了一个直接的消融实验：把ViT里的注意力层整个换掉，改用一个作用在图像块（patch）维度上的前馈网络层，整个模型就变成了交替作用于patch维度和特征维度的纯前馈层堆叠。实验结果出人意料——在ImageNet上，这个没有任何注意力机制的基础规模模型达到了74.9%的Top-1准确率，与ViT的77.9%和DeiT的79.9%差距并不悬殊，而且连超参数都没有单独调整。这说明ViT之所以强大，很可能主要来自patch嵌入（patch embedding）带来的归纳偏置以及精心设计的训练增强策略，而非注意力机制本身，这一发现提示研究者应该重新审视现有模型究竟"好在哪里"。
原文 arXiv:2105.02723;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.02723v1