Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei† Microsoft Corporation https://aka.ms/beit-3 Equal contribution. ††\dagger Corresponding author.
Abstract
A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEiT-3, which achieves state-of-the-art transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We introduce Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked “language” modeling on images (Imglish), texts (English), and image-text pairs (“parallel sentences”) in a unified manner. Experimental results show that BEiT-3 obtains state-of-the-art performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).
中文速览
把图像、文本和图文对统一到同一个模型里做预训练，一直是多模态 AI 的核心挑战。研究者提出了 BEiT-3 这个通用多模态基础模型，核心思路是把图像当作一门"外语"（Imglish），用同一套"掩码预测"（masked data modeling）任务来处理文本、图像和图文对，完全不需要对比学习等额外预训练目标。在架构上，BEiT-3 采用多路 Transformer（Multiway Transformer），通过模态专家池实现既深度融合又保留各模态特有信息的编码方式，同时将模型规模扩展到约 19 亿参数。实验结果表明，BEiT-3 在视觉问答、视觉推理、图像描述、跨模态检索、目标检测、语义分割、图像分类等几乎所有主流基准上均刷新了最优成绩，而且全程只使用公开数据，无需私有资源。这项工作的意义在于，它用极简的预训练方案证明了"一个模型、一个任务"足以统一视觉与语言领域，为后续构建更大规模的通用多模态基础模型提供了可复现的强基线。
原文 arXiv:2208.10442;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2208.10442v2