Pix2seq: A Language Modeling Framework for Object Detection
Ting Chen, Saurabh Saxena, Lala Li, David J. Fleet, Geoffrey Hinton Google Research, Brain Team Correspondence to:
Abstract
We present Pix2Seq, a simple and generic framework for object detection. Unlike existing approaches that explicitly integrate prior knowledge about the task, we cast object detection as a language modeling task conditioned on the observed pixel inputs. Object descriptions (e.g., bounding boxes and class labels) are expressed as sequences of discrete tokens, and we train a neural network to perceive the image and generate the desired sequence. Our approach is based mainly on the intuition that if a neural network knows about where and what the objects are, we just need to teach it how to read them out. Beyond the use of task-specific data augmentations, our approach makes minimal assumptions about the task, yet it achieves competitive results on the challenging COCO dataset, compared to highly specialized and well optimized detection algorithms.111Code and checkpoints available at https://github.com/google-research/pix2seq.
中文速览
目标检测领域长期依赖区域候选、边框回归、IoU损失等大量专门设计的模块，Pix2Seq把这些复杂机制彻底抛开，把检测问题转化成一个以图像为条件的序列生成（sequence generation）任务：把边框坐标和类别标签量化为离散词元，让编码器-解码器网络像做语言建模一样逐个预测这些词元，用普通的交叉熵损失训练就够了。为了解决模型容易漏检、召回率低的问题，研究者还设计了一种序列增强（sequence augmentation）技巧，在训练时往序列里掺入合成噪声对象，教模型分辨真实预测与噪声，从而在大幅提升召回率的同时抑制误检。在具有挑战性的COCO基准上，Pix2Seq的表现可与Faster R-CNN、DETR等深度定制的检测器相媲美，预训练后进一步提升；这项工作的意义在于证明了目标检测不需要任何任务专属的归纳偏置，一个通用的视觉语言框架即可胜任，为把检测能力无缝整合进多任务通用感知系统开辟了新路径。
原文 arXiv:2109.10852;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.10852v2