ResT: An Efficient Transformer for Visual Recognition
Qing-Long Zhang, Yu-Bin Yang State Key Laboratory for Novel Software Technology Nanjing University, Nanjing 21023, China This work is funded by the Natural Science Foundation of China (No. 62176119).
Abstract
This paper presents an efficient multi-scale vision Transformer, called ResT, that capably served as a general-purpose backbone for image recognition. Unlike existing Transformer methods, which employ standard Transformer blocks to tackle raw images with a fixed resolution, our ResT have several advantages: (1) A memory-efficient multi-head self-attention is built, which compresses the memory by a simple depth-wise convolution, and projects the interaction across the attention-heads dimension while keeping the diversity ability of multi-heads; (2) Positional encoding is constructed as spatial attention, which is more flexible and can tackle with input images of arbitrary size without interpolation or fine-tune; (3) Instead of the straightforward tokenization at the beginning of each stage, we design the patch embedding as a stack of overlapping convolution operation with stride on the token map. We comprehensively validate ResT on image classification and downstream tasks. Experimental results show that the proposed ResT can outperform the recently state-of-the-art backbones by a large margin, demonstrating the potential of ResT as strong backbones. The code and models will be made
中文速览
为视觉识别设计一个既高效又通用的多尺度骨干网络（backbone），是计算机视觉领域的核心挑战之一——现有Transformer方案要么计算开销随序列长度平方增长，要么只能处理固定尺寸输入，难以胜任目标检测、实例分割等密集预测任务。研究团队提出了ResT（Residual Transformer），通过三项关键改进来解决上述问题：用深度可分离卷积对键/值做空间压缩以大幅降低自注意力的内存和计算开销，同时引入跨注意力头维度的投影与实例归一化来保留多头多样性；将位置编码设计为基于局部卷积的像素级空间注意力，使模型能自然处理任意尺寸输入而无需插值；并以堆叠重叠卷积代替直接切块tokenization，以更好地提取边缘、角点等低层特征。在ImageNet-1k图像分类任务上，ResT-Small以与ResNet-18相近的参数量取得79.6%的Top-1准确率，显著优于同体量的CNN和Transformer基线，在MS COCO目标检测与实例分割任务上同样刷新了当时的最优结果，证明其作为通用视觉骨干网络的强大潜力。
原文 arXiv:2105.13677;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.13677v5