Unified-IO: A unified model for vision, language, and multi-modal tasks
Jiasen Lu†, Christopher Clark†∗, Rowan Zellers†⋄ Equal contribution. Correspondence to Roozbeh Mottaghi†⋄, Aniruddha Kembhavi†⋄ †Allen Institute for AI ⋄University of Washington Seattle
Abstract
We propose Unified-IO, a model that performs a large variety of AI tasks spanning classical computer vision tasks, including pose estimation, object detection, depth estimation and image generation, vision-and-language tasks such as region captioning and referring expression, to natural language processing tasks such as question answering and paraphrasing. Developing a single unified model for such a large variety of tasks poses unique challenges due to the heterogeneous inputs and outputs pertaining to each task, including RGB images, per-pixel maps, binary masks, bounding boxes, and language. We achieve this unification by homogenizing every supported input and output into a sequence of discrete vocabulary tokens. This common representation across all tasks allows us to train a single transformer-based architecture, jointly on over 90 diverse datasets in the vision and language fields. Unified-IO is the first model capable of performing all 7 tasks on the GRIT benchmark and produces strong results across 16 diverse benchmarks like NYUv2-Depth, ImageNet, VQA2.0, OK-VQA, Swig, VizWizGround, BoolQ, and SciTail, with no task-specific fine-tuning. Code and demos for Unified-IO are ava
中文速览
研究者们想打造一个能同时搞定海量AI任务的"万能模型"——从目标检测、深度估计、姿态估计这些经典视觉任务，到视觉问答、图像描述这些视觉语言任务，再到文本问答、摘要生成等自然语言处理任务，全部用一个模型一口气完成。核心难点在于这些任务的输入输出格式五花八门：有图片、有文字、有边界框、有逐像素深度图……为此，他们提出了Unified-IO，通过把所有输入和输出统一转化为离散词元（token）序列来打通这一壁垒：图像和密集结构用VQ-GAN编码成图像词元，边界框和关键点用坐标词元表示，文字则用常规分词处理，最终以一个标准的Transformer编码器-解码器架构在超过90个数据集上联合训练。实验结果显示，Unified-IO是首个能完成GRIT基准全部7项任务的模型，综合得分64.3，远超第二名32分，在另外16个多样化评测集上无需针对具体任务微调也表现出色。这项工作的意义在于证明了用一套统一的序列化表示可以跨越视觉与语言的模态鸿沟，为构建真正通用的多模态AI系统提供了切实可行的路径。
原文 arXiv:2206.08916;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.08916v2