On the Relationship between Self-Attention and Convolutional Layers
Jean-Baptiste Cordonnier, Andreas Loukas、Martin Jaggi École Polytechnique Fédérale de Lausanne (EPFL)
Abstract
Recent trends of incorporating attention mechanisms in vision have led researchers to reconsider the supremacy of convolutional layers as a primary building block. Beyond helping CNNs to handle long-range dependencies, Ramachandran et al. (2019) showed that attention can completely replace convolution and achieve state-of-the-art performance on vision tasks. This raises the question: do learned attention layers operate similarly to convolutional layers? This work provides evidence that attention layers can perform convolution and, indeed, they often learn to do so in practice. Specifically, we prove that a multi-head self-attention layer with sufficient number of heads is at least as expressive as any convolutional layer. Our numerical experiments then show that self-attention layers attend to pixel-grid patterns similarly to CNN layers, corroborating our analysis. Our code is publicly available111Code: github.com/epfml/attention-cnn. Website: epfml.github.io/attention-cnn..
中文速览
卷积神经网络（CNN）长期主导图像任务，而近年来研究者开始尝试用自注意力（self-attention）完全替代卷积层，并取得了不错的效果，但没人说清楚两者在计算机制上是否本质相同。这项工作从理论上严格证明：一个具有足够多头数的多头自注意力（multi-head self-attention）层，搭配相对位置编码，可以精确模拟任意卷积层——每个注意力头只需专注于卷积核中的一个固定像素偏移位置，多个头合在一起就还原出完整的卷积操作。数值实验进一步发现，在纯注意力图像分类网络中，浅层的注意力头确实自发地学出了这种"格点式"关注模式，与理论构造高度吻合，而深层则转向基于内容的注意力。这一结果揭示了局部卷积是图像浅层特征提取的天然归纳偏置，也为理解自注意力和卷积为何在视觉任务上性能相近提供了理论基础。
原文 arXiv:1911.03584;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.03584v2