AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts
Taylor Shin♢♢\>\,{}^{{\diamondsuit}} Yasaman Razeghi∗♢ Robert L. Logan IV∗♢ Eric Wallace♠ Sameer Singh♢ ♢University of California, Irvine ♠University of California, Berkeley {tshin1, yrazeghi, rlogan, First three authors contributed equally.
Abstract
The remarkable success of pretrained language models has motivated the study of what kinds of knowledge these models learn during pretraining. Reformulating tasks as fill-in-the-blanks problems (e.g., cloze tests) is a natural approach for gauging such knowledge, however, its usage is limited by the manual effort and guesswork required to write suitable prompts. To address this, we develop AutoPrompt, an automated method to create prompts for a diverse set of tasks, based on a gradient-guided search. Using AutoPrompt, we show that masked language models (MLMs) have an inherent capability to perform sentiment analysis and natural language inference without additional parameters or finetuning, sometimes achieving performance on par with recent state-of-the-art supervised models. We also show that our prompts elicit more accurate factual knowledge from MLMs than the manually created prompts on the LAMA benchmark, and that MLMs can be used as relation extractors more effectively than supervised relation extraction models. These results demonstrate that automatically generated prompts are a viable parameter-free alternative to existing probing methods, and as pretrained LMs become more
中文速览
预训练语言模型（如BERT、RoBERTa）究竟在预训练阶段学到了哪些知识，一直是个难以直接回答的问题，而手写"填空题"式提示词（prompt）来探测模型知识的方法既费时又对措辞极度敏感。为此，研究者提出了AutoPrompt——一种基于梯度搜索的自动提示词生成方法，它通过在输入文本中插入一组共享的"触发词"（trigger tokens）并迭代优化，让遮蔽语言模型（masked language model）直接输出任务预测，无需额外微调或新增参数。实验表明，经AutoPrompt提示的RoBERTa在情感分析任务SST-2上达到91%准确率（超过微调版ELMo），在事实知识检索基准LAMA上以43.3%的精确率大幅领先现有最优手写提示的34.1%，在低数据场景下还能超过微调方法的平均表现和最坏情况表现。这项工作说明，自动生成的提示词可以作为一种无需参数的探测工具，随着预训练语言模型能力持续增强，有望在更多场景下替代传统微调范式。
原文 arXiv:2010.15980;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.15980v2