Large Language Models Are Latent Variable Models: Explaining and Finding Good Demonstrations for In-Context Learning
Xinyi Wang11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT, Wanrong Zhu11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT, Michael Saxon11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT, Mark Steyvers22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT, William Yang Wang11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPTDepartment of Computer Science, University of California, Santa Barbara 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTDepartment of Cognitive Sciences, University of California, Irvine {xinyi_wang, wanrongzhu,
Abstract
In recent years, pre-trained large language models (LLMs) have demonstrated remarkable efficiency in achieving an inference-time few-shot learning capability known as in-context learning. However, existing literature has highlighted the sensitivity of this capability to the selection of few-shot demonstrations. Current understandings of the underlying mechanisms by which this capability arises from regular language model pretraining objectives remain disconnected from the real-world LLMs. This study aims to examine the in-context learning phenomenon through a Bayesian lens, viewing real-world LLMs as latent variable models. On this premise, we propose an algorithm to select optimal demonstrations from a set of annotated data with a small LM, and then directly generalize the selected demonstrations to larger LMs. We demonstrate significant improvement over baselines, averaged over eight GPT models on eight real-world text classification datasets. We also demonstrate the real-world usefulness of our algorithm on GSM8K, a math word problem dataset. Our empirical findings support our hypothesis that LLMs implicitly infer a latent variable containing task information. 111Code: https://g
中文速览
大型语言模型（LLM）的上下文学习（in-context learning）能力对示例的选择非常敏感，而现有理论研究大多停留在合成数据上，无法直接指导真实场景的实践。本文从贝叶斯视角出发，将 LLM 建模为隐变量模型，假设每个任务背后存在一个编码任务信息的隐概念变量（latent concept variable），LLM 在推理时会隐式地从示例中推断这一变量，进而做出预测。基于此，作者提出了一种两阶段示例选择算法：先用小规模语言模型（参数量不足 10 亿）通过提示调优（prompt tuning）学习出最优隐概念变量的表示，再从候选数据集中挑选最能激活该隐变量的示例，最后将这批示例直接迁移给更大的 LLM（最大达 1750 亿参数）使用。在八个真实文本分类数据集和数学应用题基准 GSM8K 上，跨越八个 GPT 系列模型的实验表明，该方法相比各类基线均有显著提升。这项工作首次将隐变量理论与真实 LLM 的上下文学习实践打通，为高效、低成本地优选示例提供了有理论依据的实用方案。
原文 arXiv:2301.11916;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.11916v4