LASP: Text-to-Text Optimization for Language-Aware Soft Prompting of Vision & Language Models
Adrian Bulat1,2, Georgios Tzimiropoulos1,3 1Samsung AI Cambridge 2Technical University of Iasi 3Queen Mary University of London
Abstract
Soft prompt learning has recently emerged as one of the methods of choice for adapting V&L models to a downstream task using a few training examples. However, current methods significantly overfit the training data, suffering from large accuracy degradation when tested on unseen classes from the same domain. To this end, in this paper, we make the following 4 contributions: (1) To alleviate base class overfitting, we propose a novel Language-Aware Soft Prompting (LASP) learning method by means of a text-to-text cross-entropy loss that maximizes the probability of the learned prompts to be correctly classified with respect to pre-defined hand-crafted textual prompts. (2) To increase the representation capacity of the prompts, we propose grouped LASP where each group of prompts is optimized with respect to a separate subset of textual prompts. (3) We identify a visual-language misalignment introduced by prompt learning and LASP, and more importantly, propose a re-calibration mechanism to address it. (4) We show that LASP is inherently amenable to including, during training, virtual classes, i.e. class names for which no visual samples are available, further increasing the robustness
中文速览
视觉-语言模型（如CLIP）在少样本场景下用可学习的"软提示词"（soft prompt）来适配新任务时，往往会严重过拟合训练类别，导致在没见过的新类上准确率大幅下滑，甚至不如手工设计的提示词。为此，本文提出了"语言感知软提示学习"（Language-Aware Soft Prompting，LASP）方法：核心思路是在原有视觉-语言训练损失之外，引入一个纯文本对文本的交叉熵损失，强制可学习提示词在文本编码空间中向手工提示词靠拢，从而把手工提示词的泛化能力"传导"给可学习提示词；此外还设计了分组提示表示来提升容量、引入层归一化微调和类无关偏置来修正视觉-语言对齐偏差，并利用纯语言优化的特性加入无图像样本的"虚拟类"进一步增强鲁棒性。在11个数据集上的实验表明，LASP全面超越此前所有软提示方法，并首次在其中8个数据集上于新类识别（零样本设置）上超越了CLIP配合手工提示词的强基线，证明可学习提示词无需依赖动态图像条件化就能真正做到对未见类的良好泛化。
原文 arXiv:2210.01115;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.01115v2