Conditional Prompt Learning for Vision-Language Models
Kaiyang Zhou Jingkang Yang Chen Change Loy Ziwei Liu Corresponding author S-Lab, Nanyang Technological University, Singapore {kaiyang.zhou, jingkang001, ccloy,
Abstract
With the rise of powerful pre-trained vision-language models like CLIP, it becomes essential to investigate ways to adapt these models to downstream datasets. A recently proposed method named Context Optimization (CoOp) introduces the concept of prompt learning—a recent trend in NLP—to the vision domain for adapting pre-trained vision-language models. Specifically, CoOp turns context words in a prompt into a set of learnable vectors and, with only a few labeled images for learning, can achieve huge improvements over intensively-tuned manual prompts. In our study we identify a critical problem of CoOp: the learned context is not generalizable to wider unseen classes within the same dataset, suggesting that CoOp overfits base classes observed during training. To address the problem, we propose Conditional Context Optimization (CoCoOp), which extends CoOp by further learning a lightweight neural network to generate for each image an input-conditional token (vector). Compared to CoOp's static prompts, our dynamic prompts adapt to each instance and are thus less sensitive to class shift. Extensive experiments show that CoCoOp generalizes much better than CoOp to unseen classes, even sho
中文速览
如何让CLIP这类视觉-语言大模型更好地泛化到没见过的类别，是当前适配预训练模型研究中的核心难题。现有方法CoOp通过把提示词（prompt）中的上下文替换为可学习向量，在少量标注数据上大幅超越手工提示，但本文发现它严重过拟合训练时见过的基础类别，一旦遇到新类别性能就大幅下滑。为此，作者提出条件上下文优化（CoCoOp），在CoOp基础上额外引入一个轻量神经网络，为每张输入图像动态生成一个条件token并融入提示向量，使提示不再是一个固定不变的静态模板，而是随每张图片自适应调整。在11个数据集上的大量实验表明，CoCoOp在新类别泛化、跨数据集迁移和域泛化三项任务上均显著优于CoOp，说明让提示依赖于具体输入实例这一简单想法，是解决提示学习泛化性不足的有效途径，也为未来构建更通用的视觉-语言模型适配方案提供了重要启示。
原文 arXiv:2203.05557;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.05557v2