Large Language Models are Human-Level Prompt Engineers
Yongchao Zhou1,2,∗, Andrei Ioan Muresanu2,3,∗, Ziwen Han1,2,∗, Keiran Paster1,2, Silviu Pitis1,2, Harris Chan1,2, Jimmy Ba1,2 1University of Toronto 2Vector Institute 3University of Waterloo ∗Equal contribution
Abstract
By conditioning on natural language instructions, large language models (LLMs) have displayed impressive capabilities as general-purpose computers. However, task performance depends significantly on the quality of the prompt used to steer the model, and most effective prompts have been handcrafted by humans. Inspired by classical program synthesis and the human approach to prompt engineering, we propose Automatic Prompt Engineer111We define “prompt engineering” as optimizing the language in a prompt in order to elicit the best possible performance. Notably, this does not include prompts that chain multiple LLM queries together or give the LLM access to external tools. (APE) for automatic instruction generation and selection. In our method, we treat the instruction as the “program,” optimized by searching over a pool of instruction candidates proposed by an LLM in order to maximize a chosen score function. To evaluate the quality of the selected instruction, we evaluate the zero-shot performance of another LLM following the selected instruction. Extensive experiments show that our automatically generated instructions outperform the prior LLM baseline by a large margin and achieve be
中文速览
如何让大语言模型（LLM）自动找到最优提示词（prompt），而不再依赖人工反复试错——这是本文要解决的核心问题。研究者提出了一套名为"自动提示工程师"（Automatic Prompt Engineer，APE）的方法：先用一个LLM根据少量输入-输出示例自动生成一批候选指令，再用评分函数筛选出表现最好的那条，必要时还通过蒙特卡洛搜索对候选指令进行语义变体迭代优化，整个流程无需人工介入。实验表明，APE生成的指令在24个Instruction Induction任务上全面达到人类水准，在21个BIG-Bench任务中的17个上与人工撰写的提示持平甚至更优，还能提升少样本学习效果、找到更好的零样本思维链（chain-of-thought）提示，并引导模型输出更真实、更具信息量的回答。这项工作意义在于，它将繁琐的提示工程自动化，让普通用户无需深厚的"调提示"经验也能充分发挥大语言模型的潜力。
原文 arXiv:2211.01910;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.01910v2