Prompt, Generate, then Cache: Cascade of Foundation Models makes Strong Few-shot Learners
Renrui Zhang∗2,3, Xiangfei Hu∗2,4, Bohao Li5, Siyuan Huang2,4, Hanqiu Deng2, Hongsheng Li3, Yu Qiao2, Peng Gao†1,2 1Shenzhen Institutes of Advanced Technology, Chinese Academy of Science 2Shanghai Artificial Intelligence Laboratory 3The Chinese University of Hong Kong 4Shanghai Jiaotong University 5University of Chinese Academy of Sciences {zhangrenrui, huangsiyuan, qiaoyu,
Abstract
Visual recognition in low-data regimes requires deep neural networks to learn generalized representations from limited training samples. Recently, CLIP-based methods have shown promising few-shot performance benefited from the contrastive language-image pre-training. We then question, if the more diverse pre-training knowledge can be cascaded to further assist few-shot representation learning. In this paper, we propose CaFo, a Cascade of Foundation models that incorporates diverse prior knowledge of various pre-training paradigms for better few-shot learning. Our CaFo incorporates CLIP’s language-contrastive knowledge, DINO’s vision-contrastive knowledge, DALL-E’s vision-generative knowledge, and GPT-3’s language-generative knowledge. Specifically, CaFo works by ‘Prompt, Generate, then Cache’. Firstly, we leverage GPT-3 to produce textual inputs for prompting CLIP with rich downstream linguistic semantics. Then, we generate synthetic images via DALL-E to expand the few-shot training data without any manpower. At last, we introduce a learnable cache model to adaptively blend the predictions from CLIP and DINO. By such collaboration, CaFo can fully unleash the potential of different
中文速览
小样本视觉识别（few-shot visual recognition）的核心难题是训练数据极度匮乏，模型很难学到足够通用的表征。针对这一问题，研究者提出了CaFo框架，将CLIP、DINO、DALL-E、GPT-3四个来自不同预训练范式的基础模型级联起来，通过"提示—生成—缓存"三步流程互补协作：先用GPT-3为CLIP生成语义更丰富的文本提示，再用DALL-E零成本合成额外训练图像扩充稀缺数据，最后用一个可学习的缓存模型自适应融合CLIP与DINO的预测结果。在11个下游数据集的小样本分类实验中，CaFo无需任何额外标注数据便达到了当时的最优性能。这项工作的重要意义在于，它首次系统性地将多种预训练范式的互补知识统一到小样本学习框架中，为如何"物尽其用"地组合现有大模型提供了清晰的思路。
原文 arXiv:2303.02151;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.02151v1