Calibrate Before Use: Improving Few-Shot Performance of Language Models
Tony Z. Zhao Eric Wallace Shi Feng Dan Klein Sameer Singh
Abstract
GPT-3 can perform numerous tasks when provided a natural language prompt that contains a few training examples. We show that this type of few-shot learning can be unstable: the choice of prompt format, training examples, and even the order of the training examples can cause accuracy to vary from near chance to near state-of-the-art. We demonstrate that this instability arises from the bias of language models towards predicting certain answers, e.g., those that are placed near the end of the prompt or are common in the pre-training data. To mitigate this, we first estimate the model’s bias towards each answer by asking for its prediction when given the training prompt and a content-free test input such as “N/A”. We then fit calibration parameters that cause the prediction for this input to be uniform across answers. On a diverse set of tasks, this contextual calibration procedure substantially improves GPT-3 and GPT-2’s average accuracy (up to 30.0% absolute) and reduces variance across different choices of the prompt.
中文速览
大语言模型（如GPT-3）在"少样本提示学习"（few-shot in-context learning）中表现飘忽不定——换一批示例、换个顺序，准确率可能从接近随机猜测（54%）跳升至接近最优水平（93%），让普通用户根本无法稳定地获得好结果。研究者发现，这种不稳定根源于模型的三类系统性偏差：偏爱提示中出现频率高的答案（多数标签偏差）、偏爱靠近提示末尾的答案（近因偏差），以及偏爱预训练数据中常见的词汇（常见词偏差）。针对这一问题，他们提出了"情境校准"（contextual calibration）方法：先把测试输入替换成无内容占位符（如"N/A"），观察模型对各答案的预测概率，再拟合一组校准参数使这种空输入下各答案的得分均匀分布，从而消除偏差。在多个分类、事实检索和信息抽取任务上，这一无需额外标注数据的简单方法让GPT-3和GPT-2的平均准确率最高提升了30个百分点，同时大幅降低了不同提示设计之间的性能方差，让少样本学习变得更可靠、更易用。
原文 arXiv:2102.09690;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.09690v2