Implicit In-context Learning
Zhuowei Li, Zihao Xu, Ligong Han, Yunhe Gao, Song Wen, Di Liu Hao Wang, Dimitris N. Metaxas Department of Computer Science Rutgers University {<first name>.<last
Abstract
In-context Learning (ICL) empowers large language models (LLMs) to swiftly adapt to unseen tasks at inference-time by prefixing a few demonstration examples before queries. Despite its versatility, ICL incurs substantial computational and memory overheads compared to zero-shot learning and is sensitive to the selection and order of demonstration examples. In this work, we introduce Implicit In-context Learning (I2CL), an innovative paradigm that reduces the inference cost of ICL to that of zero-shot learning with minimal information loss. I2CL operates by first generating a condensed vector representation, namely a context vector, extracted from the demonstration examples. It then conducts an inference-time intervention through injecting a linear combination of the context vector and query activations back into the model’s residual streams. Empirical evaluation on nine real-world tasks across three model architectures demonstrates that I2CL achieves few-shot level performance at zero-shot inference cost, and it exhibits robustness against variations in demonstration examples. Furthermore, I2CL facilitates a novel representation of “task-ids”, enhancing task similarity detection and
中文速览
上下文学习（In-context Learning，ICL）让大模型只需在输入前拼几个示例就能完成新任务，但这种做法会随示例数量急剧增加计算量和显存占用，且对示例的选择和顺序极为敏感。为此，研究者提出了隐式上下文学习（Implicit In-context Learning，I2CL）：先把所有示例各自提取一个激活向量，再取均值压缩成一个"上下文向量"，推理时只用简单的线性叠加把它注入模型的残差流，完全绕过注意力机制，从而让推理速度降至与零样本学习相同。在三种主流大模型、九个真实文本分类任务上的实验表明，I2CL以零样本的推理开销达到了少样本的准确率，同时对示例的选择和排列顺序更加稳健；此外，优化得到的少量线性系数还能自然地充当"任务标识符"，有效衡量任务相似度并辅助迁移学习，为在资源受限场景下高效部署大模型提供了切实可行的新路径。
原文 arXiv:2405.14660;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2405.14660v2