Visual Transformers: Token-based Image Representation and Processing for Computer Vision
Bichen Wu1 Chenfeng Xu3 Xiaoliang Dai1 Alvin Wan3 Peizhao Zhang1 Zhicheng Yan2 Masayoshi Tomizuka3 Joseph Gonzalez3 Kurt Keutzer3 Peter Vajda1 1 Facebook Reality Labs 2 Facebook AI 3 UC Berkeley {wbc, xiaoliangdai, stzpz, zyan3, {xuchenfeng, alvinwan, tomizuka, jegonzal,
Abstract
Computer vision has achieved remarkable success by (a) representing images as uniformly-arranged pixel arrays and (b) convolving highly-localized features. However, convolutions treat all image pixels equally regardless of importance; explicitly model all concepts across all images, regardless of content; and struggle to relate spatially-distant concepts. In this work, we challenge this paradigm by (a) representing images as semantic visual tokens and (b) running transformers to densely model token relationships. Critically, our Visual Transformer operates in a semantic token space, judiciously attending to different image parts based on context. This is in sharp contrast to pixel-space transformers that require orders-of-magnitude more compute. Using an advanced training recipe, our VTs significantly outperform their convolutional counterparts, raising ResNet accuracy on ImageNet top-1 by 4.6 to 7 points while using fewer FLOPs and parameters. For semantic segmentation on LIP and COCO-stuff, VT-based feature pyramid networks (FPN) achieve 0.35 points higher mIoU while reducing the FPN module’s FLOPs by 6.5x.
中文速览
传统卷积神经网络把图像当成均匀像素阵列处理，无论哪个区域都一视同仁，既浪费算力在背景和无关概念上，也难以捕捉远距离语义关联。这篇论文提出一种"视觉变换器"（Visual Transformer，VT）新范式：先用卷积提取低层特征，再通过空间注意力把特征图压缩成少量语义视觉词元（visual token），最后用自注意力（self-attention）变换器在这个紧凑的词元空间里建模概念间的关系，从而只把计算力集中在真正重要的内容上。实验表明，将 ResNet 最后阶段替换为 VT 后，ImageNet top-1 准确率提升 4.6 到 7 个百分点，同时 FLOPs 和参数量均更少；在语义分割任务上，VT 版特征金字塔网络（FPN）比标准 FPN 模块减少 6.5 倍计算量，mIoU 还略有提升。这项工作意义在于，它从根本上打破了像素-卷积的旧范式，用更少的词元（仅需 16 个）就超越了依赖数百个节点的图卷积方法，为高效、内容感知的视觉建模提供了一条新路径。
原文 arXiv:2006.03677;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.03677v4