CLIP-Adapter: Better Vision-Language Models with Feature Adapters
\fnmPeng \surGao \fnmShijie \surGeng \fnmRenrui \surZhang \fnmTeli \surMa \fnmRongyao \surFang \fnmYongfeng \surZhang [ [ [ [
Abstract
Large-scale contrastive vision-language pretraining has shown significant progress in visual representation learning. Unlike traditional visual systems trained by a fixed set of discrete labels, a new paradigm was introduced in Radford et al (2021) to directly learn to align images with raw texts in an open-vocabulary setting. On downstream tasks, a carefully chosen text prompt is employed to make zero-shot predictions. To avoid non-trivial prompt engineering, context optimization Zhou et al (2022) has been proposed to learn continuous vectors as task-specific prompts with few-shot training examples. In this paper, we show that there is an alternative path to achieve better vision-language models other than prompt tuning. While prompt tuning is for the textual inputs, we propose CLIP-Adapter to conduct fine-tuning with feature adapters on either visual or language branch. Specifically, CLIP-Adapter adopts an additional bottleneck layer to learn new features and performs residual-style feature blending with the original pretrained features. As a consequence, CLIP-Adapter is able to outperform context optimization while maintaining a simple design. Experiments and extensive ablation
中文速览
视觉与语言的联合预训练模型 CLIP 在零样本图像分类上表现出色，但要把它用好，要么费心设计文本提示词，要么像 CoOp 那样用少量样本去优化软提示——这两条路都绕不开对文本输入动手脚。本文提出了一条更简单的路：在 CLIP 的视觉或语言特征层后面插入一个轻量级的"适配器（Adapter）"，只用两层线性变换把原始冻结特征变换到新的表示，再通过残差连接把新旧特征按可学习的比例融合在一起，既保留了 CLIP 原有的泛化知识，又能从少量标注样本中吸收任务特定的信息。在十一个视觉分类数据集上的实验表明，CLIP-Adapter 在少样本设置下的准确率全面超过了 CoOp，而需要训练的参数量极少、设计也更加简洁。这项工作说明，对大规模视觉语言模型做轻量特征微调是提示调优之外一条切实可行的替代方案，为后续高效迁移学习研究提供了新的基线参考。
原文 arXiv:2110.04544;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.04544v2