Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generator
Hyuhng Joon Kim†, Hyunsoo Cho†, Junyeob Kim†, Taeuk Kim‡, Kang Min Yoo†§¶, Sang-goo Lee† †Seoul National University, ‡ Hanyang University, §NAVER AI Lab, ¶NAVER CLOVA {heyjoonkim, johyunsoo, juny116,
Abstract
Large-scale pre-trained language models (PLMs) are well-known for being capable of solving a task simply by conditioning a few input-label pairs dubbed demonstrations on a prompt without being explicitly tuned for the desired downstream task. Such a process (i.e., in-context learning), however, naturally leads to high reliance on the demonstrations which are usually selected from external datasets. In this paper, we propose self-generated in-context learning (SG-ICL), which generates demonstrations for in-context learning from PLM itself to minimize the reliance on the external demonstration. We conduct experiments on four different text classification tasks and show SG-ICL significantly outperforms zero-shot learning and is generally worth approximately 0.6 gold training samples. Moreover, our generated demonstrations show more consistent performance with low variance compared to randomly selected demonstrations from the training dataset.
中文速览
大型预训练语言模型做上下文学习（in-context learning，ICL）时，需要从外部数据集中挑选示例（demonstration）拼进提示词，性能因此严重依赖示例的选择质量和随机性。研究者提出了"自生成上下文学习"（SG-ICL）：让模型自己根据当前测试输入和目标类别标签来生成示例，再把这些自产的示例当作ICL的演示样本，完全绕开对外部训练数据的依赖。在四个文本分类任务上用GPT-J（6B）做实验，SG-ICL显著优于零样本学习，且8条自生成示例大约相当于0.6倍真实训练样本的价值，同时比随机选取真实示例的方差更小、性能更稳定。这项工作的意义在于，它为在没有任何标注数据的场景下使用ICL提供了一条切实可行的路径，也为如何降低ICL对外部数据依赖这一核心痛点给出了新思路。
原文 arXiv:2206.08082;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.08082v1