Prompt Engineering a Prompt Engineer
Qinyuan Ye1† Maxamed Axmed2 Reid Pryzant2 Fereshte Khani2 1University of Southern California 2Microsoft
Abstract
Prompt engineering is a challenging yet crucial task for optimizing the performance of large language models on customized tasks. It requires complex reasoning to examine the model’s errors, hypothesize what is missing or misleading in the current prompt, and communicate the task with clarity. While recent works indicate that large language models can be meta-prompted to perform automatic prompt engineering, we argue that their potential is limited due to insufficient guidance for complex reasoning in the meta-prompt. We fill this gap by infusing into the meta-prompt three key components: detailed descriptions, context specification, and a step-by-step reasoning template. The resulting method, named PE2, exhibits remarkable versatility across diverse language tasks. It finds prompts that outperform “let’s think step by step” by 6.3% on MultiArith and 3.1% on GSM8K, and outperforms competitive baselines on counterfactual tasks by 6.9%. Further, we show that PE2 can make targeted and highly specific prompt edits, rectify erroneous prompts, and induce multi-step plans for complex tasks.†††Work done while interning at Microsoft.
中文速览
让大语言模型（LLM）在特定任务上表现出色，关键在于设计出合适的提示词（prompt），但这件事既费时又需要专业经验。已有研究尝试用"元提示"（meta-prompt）让 LLM 自动优化提示词，却因元提示本身缺乏足够的推理引导而效果受限。本文提出的方法 PE2（Prompt Engineered Prompt Engineer）通过在元提示中加入三个核心要素——详细的两步骤任务描述、上下文格式说明、以及逐步推理模板——来引导 LLM 更系统地分析错误、诊断问题并改写提示词。实验表明，PE2 生成的提示词在数学推理任务上比"让我们逐步思考"基准分别提升了 6.3%（MultiArith）和 3.1%（GSM8K），在反事实任务上超越竞争基线 6.9%，还能将一个超过 5000 词元的生产级提示词的 F1 分数提升 8%。这项工作说明，对"自动提示工程师"本身进行精心的提示设计，是释放 LLM 自动优化能力的有效路径。
原文 arXiv:2311.05661;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.05661v3