In-context Vectors: Making In Context Learning More Effective and Controllable Through Latent Space Steering
Sheng Liu Haotian Ye Lei Xing James Zou
Abstract
Large language models (LLMs) demonstrate emergent in-context learning capabilities, where they adapt to new tasks based on example demonstrations. However, in-context learning has seen limited effectiveness in many settings, is difficult to quantitatively control and takes up context window space. To overcome these limitations, we propose an alternative approach that recasts in-context learning as in-context vectors (ICV). Using ICV has two steps. We first use a forward pass on demonstration examples to create the in-context vector from the latent embedding of the LLM. This vector captures essential information about the intended task. On a new query, instead of adding demonstrations to the prompt, we shift the latent states of the LLM using the ICV. The ICV approach has several benefits: 1) it enables the LLM to more effectively follow the demonstration examples; 2) it’s easy to control by adjusting the magnitude of the ICV; 3) it reduces the length of the prompt by removing the in-context demonstrations; 4) ICV is computationally much more efficient than fine-tuning. We demonstrate that ICV achieves better performance compared to standard in-context learning and fine-tuning on di
中文速览
大型语言模型（LLM）的上下文学习（in-context learning，ICL）虽然强大，却存在对示例选择敏感、占用大量上下文窗口、难以定量控制等痛点。为此，研究者提出了"上下文向量"（In-Context Vector，ICV）方法：先对示例输入和输出分别做前向传播，从模型各层的隐状态中提取出一个紧凑的任务向量；在处理新查询时，不再把示例塞进提示词，而是直接将这个向量加到模型每一层的隐状态上，把模型的表征"推向"目标方向。实验表明，ICV 在语言去毒化、风格迁移、角色扮演、格式控制等多项任务上明显优于标准 ICL 和 LoRA 微调，计算开销极小，还能通过向量加减同时组合多个任务指令。这项工作的意义在于，它提供了一种无需修改模型参数、无需长示例提示、又易于精确控制强度的轻量级任务引导范式，为 LLM 的高效个性化应用开辟了新思路。
原文 arXiv:2311.06668;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.06668v3