OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, Hongxia Yang DAMO Academy, Alibaba Group {zheluo.wp, ya235025, menrui.mr, junyang.ljy, baishuai.bs, zhikang.lzk, jason.mjx, ericzhou.zc, jingren.zhou, Correspondence to: Chang
Abstract
In this work, we pursue a unified paradigm for multimodal pretraining to break the scaffolds of complex task/modality-specific customization. We propose OFA, a Task-Agnostic and Modality-Agnostic framework that supports Task Comprehensiveness. OFA unifies a diverse set of cross-modal and unimodal tasks, including image generation, visual grounding, image captioning, image classification, language modeling, etc., in a simple sequence-to-sequence learning framework. OFA follows the instruction-based learning in both pretraining and finetuning stages, requiring no extra task-specific layers for downstream tasks. In comparison with the recent state-of-the-art vision & language models that rely on extremely large cross-modal datasets, OFA is pretrained on only $20$ M publicly available image-text pairs. Despite its simplicity and relatively small-scale training data, OFA achieves new SOTAs in a series of cross-modal tasks while attaining highly competitive performances on uni-modal tasks. Our further analysis indicates that OFA can also effectively transfer to unseen tasks and unseen domains. Our code and models are publicly available at https://github.com/OFA-Sys/OFA.
中文速览
多模态AI模型长期面临一个痛点：不同任务、不同模态往往需要单独定制模型结构和训练目标，既繁琐又难以泛化。OFA（One For All）提出了一套统一的序列到序列（sequence-to-sequence）框架，用同一个Transformer编解码器、同一套词表和基于指令（instruction）的输入格式，把图像生成、视觉定位、图像描述、视觉问答、图像分类、语言建模等十余种跨模态与单模态任务全部统一处理，无需为下游任务添加任何额外的任务专属模块。仅用2000万公开图文对预训练，OFA在多项跨模态基准上达到新的最优性能，在纯文本和纯视觉任务上也能与专门的单模态顶尖模型相媲美，还展现出对未见任务和未见领域的零样本迁移能力。这项工作表明，通过足够统一的架构设计与任务表示，无需海量私有数据也能训练出兼顾多模态理解与生成的通用模型，为构建真正"一模型打天下"的AI系统提供了切实可行的路径。
原文 arXiv:2202.03052;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.03052v2