Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Training
Anthony Meng Huat Tiong1,2, Junnan Li1, Boyang Li2, Silvio Savarese1, and Steven C.H. Hoi1 1Salesforce Research 2Nanyang Technological University, Singapore {anthony.tiong, junnan.li, ssavarese, https://github.com/salesforce/LAVIS/tree/main/projects/pnp-vqa
Abstract
Visual question answering (VQA) is a hallmark of vision and language reasoning and a challenging task under the zero-shot setting. We propose Plug-and-Play VQA (PnP-VQA), a modular framework for zero-shot VQA. In contrast to most existing works, which require substantial adaptation of pretrained language models (PLMs) for the vision modality, PnP-VQA requires no additional training of the PLMs. Instead, we propose to use natural language and network interpretation as an intermediate representation that glues pretrained models together. We first generate question-guided informative image captions, and pass the captions to a PLM as context for question answering. Surpassing end-to-end trained baselines, PnP-VQA achieves state-of-the-art results on zero-shot VQAv2 Goyal et al. (2017) and GQA Hudson and Manning (2019). With 11B parameters, it outperforms the 80B-parameter Flamingo model Alayrac et al. (2022) by 8.5% on VQAv2. With 738M PLM parameters, PnP-VQA achieves an improvement of 9.1% on GQA over FewVLM Jin et al. (2022) with 740M PLM parameters.
中文速览
视觉问答（Visual Question Answering, VQA）要求模型同时理解图片和问题并给出答案，在零样本（zero-shot）场景下尤为困难，因为模型无法依赖任务专属的训练数据。研究者提出了"即插即用视觉问答"框架（Plug-and-Play VQA, PnP-VQA），核心思路是用自然语言描述（图像标题）作为视觉与语言两个预训练模型之间的桥梁——先借助网络可解释性技术（GradCAM）找出图像中与问题最相关的区域，再对这些区域随机采样生成多条多样化的描述性标题，最后将这些标题连同问题一起送入纯文本语言模型完成作答，全程无需对任何预训练模型做额外训练或架构改动。实验表明，该框架在零样本VQAv2和GQA基准上均达到当时最优水平：仅用110亿参数就比拥有800亿参数、经过大规模视觉-语言预训练的Flamingo模型高出8.5%。这项工作证明了将多个大规模预训练模型像积木一样组合起来、无需端到端联合训练也能取得优异效果的可行性，为构建灵活、可扩展的通用多模态AI系统提供了一条新路径。
原文 arXiv:2210.08773;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.08773v3