Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions
Wenhai Wang1, Enze Xie2, Xiang Li3, Deng-Ping Fan4🖂, Kaitao Song3, Ding Liang5, Tong Lu1🖂, Ping Luo2, Ling Shao4 1Nanjing University 2The University of Hong Kong 3Nanjing University of Science and Technology 4IIAI 5SenseTime Research https://github.com/whai362/PVT
Abstract
Although convolutional neural networks (CNNs) have achieved great success in computer vision, this work investigates a simpler, convolution-free backbone network useful for many dense prediction tasks. Unlike the recently-proposed Vision Transformer (ViT) that was designed for image classification specifically, we introduce the Pyramid Vision Transformer (PVT)††🖂 Corresponding authors: Deng-Ping Fan (dengpfan@gmail.com); Tong Lu (lutong@nju.edu.cn)., which overcomes the difficulties of porting Transformer to various dense prediction tasks. PVT has several merits compared to current state of the arts. (1) Different from ViT that typically yields low-resolution outputs and incurs high computational and memory costs, PVT not only can be trained on dense partitions of an image to achieve high output resolution, which is important for dense prediction, but also uses a progressive shrinking pyramid to reduce the computations of large feature maps. (2) PVT inherits the advantages of both CNN and Transformer, making it a unified backbone for various vision tasks without convolutions, where it can be used as a direct replacement for CNN backbones. (3) We validate PVT through extensive expe
中文速览
卷积神经网络在视觉任务中虽然表现出色，但为密集预测（如目标检测、语义分割）设计一个无卷积的纯Transformer骨干网络仍是空白——已有的Vision Transformer（ViT）只能输出单尺度低分辨率特征图，计算开销也很大。为此，作者提出了金字塔视觉Transformer（Pyramid Vision Transformer，PVT），通过"渐进收缩金字塔"将图像分成细粒度小块（4×4像素）并逐阶段压缩特征图分辨率，同时引入空间压缩注意力（Spatial-Reduction Attention，SRA）降低高分辨率阶段的计算与内存消耗，从而在保持全局感受野的同时生成多尺度特征图。实验结果表明，PVT可作为ResNet等CNN骨干的直接替代品，在目标检测、实例分割和语义分割上全面超越同参数量的ResNet/ResNeXt基线，例如PVT-Small搭配RetinaNet在COCO上达到40.4 AP，比ResNet50高出4.1个点。这项工作的意义在于，它首次验证了纯Transformer骨干网络在密集预测任务中的可行性，并与DETR结合构成了首个完全无卷积、无手工后处理的端到端检测系统，为后续视觉Transformer研究奠定了重要基础。
原文 arXiv:2102.12122;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.12122v2