Demystifying Prompts in Language Models via Perplexity Estimation
Hila Gonen1,2 Srini Iyer2 Terra Blevins1 Noah A. Smith1,3 Luke Zettlemoyer1,2 1Paul G. Allen School of Computer Science、Engineering, University of Washington 2Meta AI Research 3Allen Institute for Artificial Intelligence
Abstract
Language models can be prompted to perform a wide variety of tasks with zero- and few-shot in-context learning. However, performance varies significantly with the choice of prompt, and we do not yet understand why this happens. In this paper, we analyze the factors that contribute to this variance and establish a new empirical hypothesis: the performance of a prompt is predicted by the extent to which the model is familiar with the language it contains. Over a wide range of tasks, we show that the lower the perplexity of the prompt, the better it is able to perform the task, when considering reasonable prompts that are related to it. As part of our analysis, we also devise a method to automatically extend a small seed set of manually written prompts by paraphrasing with GPT3 and backtranslation. This larger set allows us to verify that perplexity is a strong predictor of the success of a prompt and we show that the lowest perplexity prompts are consistently effective.
中文速览
提示词（prompt）的措辞差异会让同一个语言模型在同一任务上的准确率相差30个百分点以上，但我们至今不清楚背后的原因。研究者提出了一个直觉上很自然的假说：模型对提示词越"熟悉"——即提示词在预训练数据中以各种形式出现得越频繁——它就越能借助这条提示完成任务；由于直接检索训练数据既不现实又常常不可行，他们用模型对提示词计算的困惑度（perplexity）来替代衡量这种熟悉程度。为了充分验证这一假说，他们还设计了一套自动扩充提示词库的方法：先人工写少量种子提示，再用GPT-3做同义改写、用多语言回译进一步扩展，从而在多个分类和词预测任务上积累了上百条风格各异的提示词。实验横跨OPT和Bloom等不同规模模型，结果一致表明困惑度与任务表现之间存在显著负相关——困惑度越低的提示词，任务得分越高；据此提出的自动选提示方法SPELL在无需任何标注数据的情况下，平均准确率分别提升了约1.8和2.3个百分点。这项工作不仅为"为什么有些提示词更好"给出了可量化的解释，也为非专业用户提供了一套无需标注、开箱即用的提示优化工具。
原文 arXiv:2212.04037;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.04037v2