In-context Example Selection with Influences
Tai Nguyen University of Pennsylvania Eric Wong University of Pennsylvania
Abstract
In-context learning (ICL) is a powerful paradigm emerged from large language models (LLMs). Despite its promises, ICL performance is known to be highly sensitive to input examples. In this work, we use in-context influences to analyze few-shot ICL performance directly from the in-context examples. Our proposed influence-based example selection method can identify both positive and negative examples, outperforming several baselines when evaluated on 9 SuperGLUE tasks. Our analysis uncovers up to a $16.3\%$ performance gap between using the most negative in-context examples compared to the most positive. In a case study, we apply our influence-based framework to quantify the phenomena of recency bias in example ordering for few-shot ICL.111Our code is released at https://github.com/DebugML/incontext_influences.
中文速览
大规模语言模型（LLM）做少样本上下文学习（in-context learning, ICL）时，选哪几个示例放进提示词对最终效果影响极大，但究竟哪些示例"有用"、哪些"有害"，此前缺乏一套直接量化的方法。研究者借鉴数据影响力（data influence）的思路，提出"上下文影响力"框架：随机抽取训练样例子集构成提示词、测量验证集准确率，再用均值差估计每个示例对ICL性能的贡献，整个过程只需模型前向推理、无需任何梯度更新。在8个语言模型和9个SuperGLUE任务上的实验表明，基于影响力得分选出的正向示例比负向示例最多可带来16.3%的性能差距，且在正向和负向示例筛选上均优于语义相似度、困惑度等多种基线方法。该框架还可灵活拓展到其他分析场景，论文将其用于量化ICL中"近因偏差"（recency bias）对示例排序的影响，为理解和改善ICL的不稳定性提供了一个通用、低成本的工具。
原文 arXiv:2302.11042;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2302.11042v2