PromptCap: Prompt-Guided Task-Aware Image Captioning
Yushi Hu1 Hang Hua2∗ Zhengyuan Yang3 Weijia Shi1 Noah A. Smith1,4 Jiebo Luo2 1University of Washington 2University of Rochester 3Microsoft 4Allen Institute for AI https://yushi-hu.github.io/promptcap_demo/ Equal contribution. Correspondance to <Yushi Hu: <Hang Hua:
Abstract
Knowledge-based visual question answering (VQA) involves questions that require world knowledge beyond the image to yield the correct answer. Large language models (LMs) like GPT-3 are particularly helpful for this task because of their strong knowledge retrieval and reasoning capabilities. To enable LM to understand images, prior work uses a captioning model to convert images into text. However, when summarizing an image in a single caption sentence, which visual entities to describe are often underspecified. Generic image captions often miss visual details essential for the LM to answer visual questions correctly. To address this challenge, we propose PromptCap (Prompt-guided image Captioning), a captioning model designed to serve as a better connector between images and black-box LMs. Different from generic captions, PromptCap takes a natural-language prompt to control the visual entities to describe in the generated caption. The prompt contains a question that the caption should aid in answering. To avoid extra annotation, PromptCap is trained by examples synthesized with GPT-3 and existing datasets. We demonstrate PromptCap’s effectiveness on an existing pipeline in which GPT-
中文速览
基于视觉的知识问答（Knowledge-based VQA）要求模型不仅看懂图片，还得结合世界知识推理出答案，而主流做法是先用图像描述模型把图片转成文字，再交给 GPT-3 这类大语言模型（Large Language Model）去回答，但通用描述往往漏掉解题关键细节，比如图中招牌写的是"麦当劳"。为此，研究者提出了 PromptCap——一个以问题为引导的图像描述模型，输入图片的同时接收当前问题作为提示，从而生成专门包含解题所需视觉细节的描述；由于没有现成的"问题感知描述"标注数据，他们设计了一套用 GPT-3 自动合成并过滤训练样本的流水线，完全无需人工额外标注。实验表明，PromptCap 在 OK-VQA 和 A-OKVQA 两个知识问答基准上达到当时最优水平（分别为 60.4% 和 59.6%），比同架构的通用描述模型高出 3–9 个百分点，并在未见过的 WebQA 数据集上也展现出良好的泛化能力。这项工作的重要意义在于，它提供了一种轻量、无需访问模型内部参数的方式，让纯文本大模型也能"读懂"图片中与问题真正相关的信息，为黑盒大模型与视觉任务的结合提供了实用新思路。
原文 arXiv:2211.09699;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.09699v4