SimVLM: Simple Visual Language Model Pretraining with Weak Supervision
Zirui Wang1,2, Jiahui Yu2, Adams Wei Yu2, Zihang Dai2, Yulia Tsvetkov3, Yuan Cao2 1Carnegie Mellon University 2Google Research, Brain Team 3University of Washington This work was conducted at Google.
Abstract
With recent progress in joint modeling of visual and textual representations, Vision-Language Pretraining (VLP) has achieved impressive performance on many multimodal downstream tasks. However, the requirement for expensive annotations including clean image captions and regional labels limits the scalability of existing approaches, and complicates the pretraining procedure with the introduction of multiple dataset-specific objectives. In this work, we relax these constraints and present a minimalist pretraining framework, named Simple Visual Language Model (SimVLM). Unlike prior work, SimVLM reduces the training complexity by exploiting large-scale weak supervision, and is trained end-to-end with a single prefix language modeling objective. Without utilizing extra data or task-specific customization, the resulting model significantly outperforms previous pretraining methods and achieves new state-of-the-art results on a wide range of discriminative and generative vision-language benchmarks, including VQA (+3.74% vqa-score), NLVR2 (+1.17% accuracy), SNLI-VE (+1.37% accuracy) and image captioning tasks (+10.1% average CIDEr score). Furthermore, we demonstrate that SimVLM acquires str
中文速览
现有视觉-语言预训练（Vision-Language Pretraining, VLP）模型普遍依赖昂贵的人工标注数据（如目标检测标签、干净图文对），还需要堆叠多个复杂的辅助训练目标，既费钱又难以扩展。SimVLM提出了一套极简方案：直接用网络上爬取的海量弱标注图文对，以单一的"前缀语言建模（Prefix Language Modeling, PrefixLM）"目标端到端训练一个Transformer模型，图像被当作文本描述的"前缀"输入，完全抛弃了目标检测模块和各类辅助损失。实验结果显示，SimVLM在VQA、图像描述生成、视觉推理等六项主流基准上均刷新了当时的最优成绩，图像描述任务的CIDEr分数平均提升超过10%；更重要的是，该模型还展现出零样本（zero-shot）泛化能力，能够在未见过配对数据的情况下完成开放式视觉问答和跨模态迁移。这项工作表明，用生成式语言建模目标进行多模态预训练，可以同时兼顾判别性任务的强大性能与类GPT-3的零样本泛化潜力，为后续大规模多模态基础模型的设计提供了重要参考。
原文 arXiv:2108.10904;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2108.10904v3