Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models
Lei Wang1 Wanyu Xu2 Yihuai Lan Zhiqiang Hu3 Yunshi Lan4 Roy Ka-Wei Lee3 Ee-Peng Lim1 1Singapore Management University 2Southwest Jiaotong University 3Singapore University of Technology and Design 4East China Normal University Corresponding author.
Abstract
Large language models (LLMs) have recently been shown to deliver impressive performance in various NLP tasks. To tackle multi-step reasoning tasks, few-shot chain-of-thought (CoT) prompting includes a few manually crafted step-by-step reasoning demonstrations which enable LLMs to explicitly generate reasoning steps and improve their reasoning task accuracy. To eliminate the manual effort, Zero-shot-CoT concatenates the target problem statement with “Let’s think step by step” as an input prompt to LLMs. Despite the success of Zero-shot-CoT, it still suffers from three pitfalls: calculation errors, missing-step errors, and semantic misunderstanding errors. To address the missing-step errors, we propose Plan-and-Solve (PS) Prompting. It consists of two components: first, devising a plan to divide the entire task into smaller subtasks, and then carrying out the subtasks according to the plan. To address the calculation errors and improve the quality of generated reasoning steps, we extend PS prompting with more detailed instructions and derive PS+ prompting. We evaluate our proposed prompting strategy on ten datasets across three reasoning problems. The experimental results over GPT-3
中文速览
让大型语言模型（LLM）在不提供任何示例的情况下准确完成多步推理，一直是个棘手问题——现有的零样本思维链方法（Zero-shot Chain-of-Thought, CoT）虽然只需在题目后面加一句"Let's think step by step"就能触发推理，但仍频繁出现计算出错、推理步骤遗漏和语义理解偏差三类问题。为此，研究者提出了"计划与求解"提示法（Plan-and-Solve Prompting, PS），核心思路是把原来那句触发语替换成"先理解题意、制定计划，再按计划逐步求解"，让模型主动把大问题拆成小任务；在此基础上进一步加入"提取关键变量及数值""注意计算过程""计算中间结果"等细化指令，形成更强的PS+版本。在涵盖算术、常识和符号推理共十个数据集上用GPT-3测试，PS+在所有数据集上都大幅超越Zero-shot CoT，在多数数据集上还持平甚至超过需要手写示例的少样本CoT方法，却完全不需要人工标注任何演示样例。这项工作表明，仅通过精心设计零样本提示语就能显著提升LLM的推理质量，为降低复杂推理任务的人工成本提供了一条切实可行的路径。
原文 arXiv:2305.04091;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.04091v3