X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
Yan Zeng ByteDance Research、Xinsong Zhang ByteDance Research、Hang Li ByteDance Research Jiawei Wang ByteDance Research、Jipeng Zhang HKUST、Wangchunshu Zhou ETH Zurich Correspondence to:
Abstract
Vision language pre-training aims to learn alignments between vision and language from a large amount of data. Most existing methods only learn image-text alignments. Some others utilize pre-trained object detectors to leverage vision language alignments at the object level. In this paper, we propose to learn multi-grained vision language alignments by a unified pre-training framework that learns multi-grained aligning and multi-grained localization simultaneously. Based on it, we present X2-VLM, an all-in-one model with a flexible modular architecture, in which we further unify image-text pre-training and video-text pre-training in one model. X2-VLM is able to learn unlimited visual concepts associated with diverse text descriptions. Experiment results show that X2-VLM performs the best on base and large scale for both image-text and video-text tasks, making a good trade-off between performance and model scale. Moreover, we show that the modular design of X2-VLM results in high transferability for it to be utilized in any language or domain. For example, by simply replacing the text encoder with XLM-R, X2-VLM outperforms state-of-the-art multilingual multi-modal pre-trained models
中文速览
多粒度视觉语言对齐一直是个难题：粗粒度方法只学整图与文本的匹配，细粒度方法依赖目标检测器但只能识别有限类别、且无法捕捉物体间关系。X2-VLM提出了一个统一预训练框架，用视觉Transformer同时学习物体、区域和整图三个粒度的视觉概念与文本描述之间的对齐，并在同一框架内把图文预训练和视频文本预训练合并到一个模型里。实验表明，X2-VLM在图文和视频文本任务上均达到同等规模模型的最优水平，约5.9亿参数的大版本在检索、视觉问答、推理等任务上可与参数量达20亿的模型相抗衡；更令人惊喜的是，凭借模块化架构，只需把文本编码器换成多语言模型XLM-R，无需任何多语言多模态预训练，便能超越现有多语言多模态方法。这项工作表明，多粒度对齐与模块化设计的结合，是构建高效、可迁移、跨语言视觉语言模型的一条可行路径。
原文 arXiv:2211.12402;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.12402v2