BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models
Junnan Li Dongxu Li Silvio Savarese Steven Hoi Salesforce Research
Abstract
The cost of vision-and-language pre-training has become increasingly prohibitive due to end-to-end training of large-scale models. This paper proposes BLIP-2, a generic and efficient pre-training strategy that bootstraps vision-language pre-training from off-the-shelf frozen pre-trained image encoders and frozen large language models. BLIP-2 bridges the modality gap with a lightweight Querying Transformer, which is pre-trained in two stages. The first stage bootstraps vision-language representation learning from a frozen image encoder. The second stage bootstraps vision-to-language generative learning from a frozen language model. BLIP-2 achieves state-of-the-art performance on various vision-language tasks, despite having significantly fewer trainable parameters than existing methods. For example, our model outperforms Flamingo80B by 8.7% on zero-shot VQAv2 with 54x fewer trainable parameters. We also demonstrate the model’s emerging capabilities of zero-shot image-to-text generation that can follow natural language instructions.
中文速览
训练多模态大模型代价越来越高，根本原因是图像编码器和语言模型需要一起端到端微调。BLIP-2 的思路是把两个已经训练好的"冻结"单模态模型——视觉编码器（如 CLIP ViT）和大语言模型（如 OPT、FlanT5）——直接拿来用，中间只插入一个轻量级的"查询变换器"（Querying Transformer，Q-Former）来负责跨模态对齐；Q-Former 分两阶段训练：第一阶段让它从冻结的图像编码器里学会提取与文本最相关的视觉特征，第二阶段把它的输出接到冻结的大语言模型上，让语言模型学会"读懂"这些视觉特征。实验结果显示，BLIP-2 在零样本视觉问答（VQAv2）上比 Flamingo 80B 高出 8.7 个百分点，而可训练参数量仅为后者的 1/54，还能跟随自然语言指令做图文生成、视觉推理等新兴任务。这项工作的意义在于，它提供了一条"即插即用"的多模态预训练范式——未来每当视觉或语言领域出现更强的基础模型，只需替换对应的冻结模块并重新训练这个轻量桥接器，就能低成本地获得更强的多模态能力。
原文 arXiv:2301.12597;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.12597v3