A Unified Sequence Interface for Vision Tasks
Ting Chen† Saurabh Saxena† Lala Li† Tsung-Yi Lin∗ David J. Fleet Geoffrey Hinton Google Research, Brain Team †Equal contributions. ∗Work done at Google.
Abstract
While language tasks are naturally expressed in a single, unified, modeling framework, i.e., generating sequences of tokens, this has not been the case in computer vision. As a result, there is a proliferation of distinct architectures and loss functions for different vision tasks. In this work we show that a diverse set of “core” computer vision tasks can also be unified if formulated in terms of a shared pixel-to-sequence interface. We focus on four tasks, namely, object detection, instance segmentation, keypoint detection, and image captioning, all with diverse types of outputs, e.g., bounding boxes or dense masks. Despite that, by formulating the output of each task as a sequence of discrete tokens with a unified interface, we show that one can train a neural network with a single model architecture and loss function on all these tasks, with no task-specific customization. To solve a specific task, we use a short prompt as task description, and the sequence output adapts to the prompt so it can produce task-specific output. We show that such a model can achieve competitive performance compared to well-established task-specific models.
中文速览
用一个统一的"像素转序列"框架同时解决目标检测、实例分割、关键点检测和图像描述四类视觉任务，是本文的核心目标。作者将每类任务的输出（边界框坐标、多边形掩码点、关键点坐标、文字描述）全部离散化为词元序列，配合一段简短的任务提示词，让同一个编码器-解码器网络在统一的交叉熵损失下学习所有任务，无需为每项任务设计专属网络头或损失函数。在标准 COCO 基准上，该模型在四个任务上均取得了与专门模型相当的竞争性性能，且只需改变提示词即可切换任务。这项工作表明，"为每个视觉任务单独设计架构"的惯例并非必要，统一序列接口有望成为迈向通用视觉模型的可行路径。
原文 arXiv:2206.07669;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.07669v2