Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models
Robert L. Logan IV1 Ivana Balažević22\>\,{}^{2} Eric Wallace3 Fabio Petroni4 Sameer Singh1 Sebastian Riedel4,5 1UC Irvine 2University of Edinburgh 3UC Berkeley 4Facebook AI Research 5University College London Work done while an intern at Facebook AI Research.
Abstract
Prompting language models (LMs) with training examples and task descriptions has been seen as critical to recent successes in few-shot learning. In this work, we show that finetuning LMs in the few-shot setting can considerably reduce the need for prompt engineering. In fact, one can use null prompts, prompts that contain neither task-specific templates nor training examples, and achieve competitive accuracy to manually-tuned prompts across a wide range of tasks. While finetuning LMs does introduce new parameters for each downstream task, we show that this memory overhead can be substantially reduced: finetuning only the bias terms can achieve comparable or better accuracy than standard finetuning while only updating 0.1% of the parameters. All in all, we recommend finetuning LMs for few-shot learning as it is more accurate, robust to different prompts, and can be made nearly as efficient as using frozen LMs.
中文速览
少样本学习场景下，精心设计提示词（prompt engineering）往往费时费力且难以复现，研究者们希望找到一种既简单又有效的替代方案。这项工作发现，只要对语言模型进行微调（finetuning），提示词的设计可以极大简化——甚至可以用"空提示"（null prompt），即仅将输入文本与 [MASK] 标记拼接，不加任何人工设计的模板或示例，就能在多项分类任务上与精心手工调出的提示词打平。为了解决微调带来的每个任务都要存储一套完整模型参数的内存开销问题，作者进一步验证了只更新模型偏置项（BitFit）这一轻量微调方案，仅需调整全部参数的 0.1%，即可达到与全量微调相当甚至更好的准确率。这一结论对实际应用意义重大：它表明少样本学习不必依赖复杂的提示工程，微调后的模型在准确性、对提示变化的鲁棒性以及参数效率三方面均优于依赖固定模型的上下文学习（in-context learning），为工业界低成本部署提供了更可行的路径。
原文 arXiv:2106.13353;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.13353v2