Prompt Waywardness: The Surprising Power and Risks of Continuous Prompts
Daniel Khashabi‡ Xinxi Lyu† Sewon Min† Lianhui Qin† Kyle Richardson‡ Sean Welleck†‡ Hannaneh Hajishirzi†‡ Tushar Khot‡ Ashish Sabharwal‡ Sameer Singh‡♡ Yejin Choi†‡ †University of Washington ‡Allen Institute for AI ♡University of California-Irvine
Abstract
Fine-tuning continuous prompts for target tasks has recently emerged as a compact alternative to full model fine-tuning. Motivated by these promising results, we investigate the feasibility of extracting a discrete (textual) interpretation of continuous prompts that is faithful to the problem they solve. In practice, we observe a “wayward” behavior between the task solved by continuous prompts and the nearest neighbor discrete projections of these prompts: One can find continuous prompts that solve a task while being projected to an arbitrary text (e.g., definition of a different or even a contradictory task) and simultaneously being within a very small (2%) margin of the best continuous prompt of the same size for the task. We provide intuitions behind this odd and surprising behavior, as well as extensive empirical analyses quantifying the effect of design choices. For instance, larger models exhibit higher waywardness, i.e, we can find prompts that more closely map to any arbitrary text with a smaller drop of accuracy. These findings have important implications relating to the difficulty of faithfully interpreting continuous prompts and their generalization across models and tas
中文速览
连续提示（continuous prompt）作为一种比全量微调更紧凑的方案正受到广泛关注，但人们始终希望知道这些由实数向量构成的提示究竟"在说什么"——能否把它们翻译成人类可读的离散文本来解释其行为。研究者提出并验证了"提示任性（Prompt Waywardness）"假说：通过在优化目标中同时最小化任务损失和提示与目标文本的距离，可以找到这样的连续提示——它在情感分类等任务上的准确率与最优连续提示相差不超过2%，但将其投影到最近邻离散词表时，却完整映射到一段毫不相关甚至相互矛盾的文本（例如"翻转情感"的定义），两者之间几乎没有语义对应关系。在五个分类数据集和60余种目标文本上的大量实验进一步表明，模型越大、提示越长，这种"任性"现象就越严重。这一发现揭示了连续提示在可解释性上的根本局限：仅靠最近邻投影无法忠实反映提示的实际功能，同时也警示研究者，离散提示与连续提示的优化结果并不能随意互换，恶意攻击者甚至可以借此将有害提示伪装成看似无害的文本。
原文 arXiv:2112.08348;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.08348v2