In-context Learning with Retrieved Demonstrations for Language Models: A Survey
Man Luo11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Xin Xu22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Yue Liu22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Panupong Pasupat22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT Mehran Kazemi22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT 1 Arizona State University 2 Google Research {xxujasmine, yliujune, ppasupat,
Abstract
Language models, especially pre-trained large language models, have showcased remarkable abilities as few-shot in-context learners (ICL), adept at adapting to new tasks with just a few demonstrations in the input context. However, the model’s ability to perform ICL is sensitive to the choice of the few-shot demonstrations. Instead of using a fixed set of demonstrations, one recent development is to retrieve demonstrations tailored to each input query. The implementation of demonstration retrieval is relatively straightforward, leveraging existing databases and retrieval systems. This not only improves the efficiency and scalability of the learning process but also has been shown to reduce biases inherent in manual example selection. In light of the encouraging results and growing research in ICL with retrieved demonstrations, we conduct an extensive review of studies in this area. In this survey, we discuss and compare different design choices for retrieval models, retrieval training procedures, and inference algorithms.
中文速览
大型语言模型具备"看几个例子就能学会新任务"的上下文学习（in-context learning，ICL）能力，但示例的选择对效果影响极大——用固定示例往往不够好，而针对每条输入动态检索最合适的示例（retrieval-based ICL，RetICL）则能显著提升性能。这篇综述系统梳理了22篇RetICL核心论文，从检索模型的设计、训练方式到推理策略，逐一对比分析了各类方法的异同与优劣。研究发现，检索式示例不仅持续优于随机或人工选定的示例，还能降低模型对示例顺序等敏感因素的依赖，同时减少人工选例带来的偏差。这篇综述填补了该领域术语混乱、方法分散的空白，既为初学者提供了系统入门路径，也为研究者指出了未来值得深挖的方向。
原文 arXiv:2401.11624;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2401.11624v5