Toward Transformer-Based Object Detection
Josh Beal Eric Kim Eric Tzeng Dong Huk Park Andrew Zhai Dmitry Kislyuk Pinterest {jbeal, erickim, etzeng, dhukpark, andrew, Authors contributed equally.
Abstract
Transformers have become the dominant model in natural language processing, owing to their ability to pretrain on massive amounts of data, then transfer to smaller, more specific tasks via fine-tuning. The Vision Transformer was the first major attempt to apply a pure transformer model directly to images as input, demonstrating that as compared to convolutional networks, transformer-based architectures can achieve competitive results on benchmark classification tasks. However, the computational complexity of the attention operator means that we are limited to low-resolution inputs. For more complex tasks such as detection or segmentation, maintaining a high input resolution is crucial to ensure that models can properly identify and reflect fine details in their output. This naturally raises the question of whether or not transformer-based architectures such as the Vision Transformer are capable of performing tasks other than classification. In this paper, we determine that Vision Transformers can be used as a backbone by a common detection task head to produce competitive COCO results. The model that we propose, ViT-FRCNN, demonstrates several known properties associated with trans
中文速览
把视觉变换器（Vision Transformer，ViT）这类纯注意力机制模型用于目标检测是一大挑战——它原本只做图像分类，而检测任务需要模型保留精细的空间位置信息。作者提出 ViT-FRCNN，将 ViT 输出的空间 patch 特征直接接入 Faster R-CNN 检测头，并通过双线性插值位置编码来支持高分辨率、任意长宽比的输入，从而把 ViT 变成一个可用于检测的骨干网络。实验表明，ViT-FRCNN 在 COCO 基准上取得了与 ResNet-FPN 相当甚至更优的成绩，并在大型物体检测、域外图像泛化以及减少重复预测框等方面表现出明显优势，且大规模预训练数据越多，迁移效果越好。这项工作证明了纯变换器架构不仅能做分类，同样具备完成目标检测等复杂视觉任务的潜力，为后续"全变换器"视觉模型铺平了道路。
原文 arXiv:2012.09958;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.09958v1