Exploring Effective Factors for Improving Visual In-Context Learning
Yanpeng Sun1,2, Qiang Chen1, Xiaofan Li1, Jian Wang1, Jingdong Wang1, Zechao Li2 1Baidu VIS 2School of Computer Science and Engineering, Nanjing University of Science and Technology Corresponding author.
Abstract
The In-Context Learning (ICL) is to understand a new task via a few demonstrations (aka. prompt) and predict new inputs without tuning the models. While it has been widely studied in NLP, it is still a relatively new area of research in computer vision. To reveal the factors influencing the performance of visual in-context learning, this paper shows that prompt selection and prompt fusion are two major factors that have a direct impact on the inference performance of visual context learning. Prompt selection is the process of identifying the most appropriate prompt or example to help the model understand new tasks. This is important because providing the model with relevant prompts can help it learn more effectively and efficiently. Prompt fusion involves combining knowledge from different positions within the large-scale visual model. By doing this, the model can leverage the diverse knowledge stored in different parts of the model to improve its performance on new tasks. Based these findings, we propose a simple framework prompt-SelF for visual in-context learning. Specifically, we first use the pixel-level retrieval method to select a suitable prompt, and then use different prom
中文速览
视觉大模型的"上下文学习"（In-Context Learning，ICL）能力令人期待——只需提供少量示例（提示），模型无需微调就能应对新任务，但究竟哪些因素决定了它的表现好坏，此前几乎没有系统研究。本文发现，提示选择（选哪张示例图）和提示融合（怎么把示例图和查询图拼在一起输入模型）是影响视觉ICL性能的两大关键因素，前者决定了示例与查询图的相关性，后者决定了激活大模型内部哪部分知识。基于此，作者提出了prompt-SelF框架：先用像素级检索找到与查询图视觉内容最相似的提示图，再用八种不同的图像拼接排列分别调用模型的多样知识，最后通过投票集成八种预测结果得到最终输出。在单目标分割和检测任务上的实验表明，prompt-SelF首次让视觉ICL方法在单样本分割上超越了专为此设计的元学习方法OSLSM，证明了视觉上下文学习的巨大潜力。
原文 arXiv:2304.04748;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.04748v2