Making Large Language Models Better Reasoners with Step-Aware Verifier
Yifei Li1,21,2{}^{\textbf{1,2}}, Zeqi Lin22{}^{\textbf{2}}, Shizhuo Zhang22{}^{\textbf{2}}, Qiang Fu22{}^{\textbf{2}}, Bei Chen22{}^{\textbf{2}}, Jian-Guang Lou22{}^{\textbf{2}}, Weizhu Chen22{}^{\textbf{2}} 1 National Key Laboratory for Multimedia Information Processing, School of Computer Science, Peking University 2 Microsoft Corporation {yifeili, zeqi.lin, v-shizzhang, qifu, bei.chen, jlou, Work was done during an internship at Microsoft Research Asia.
Abstract
Few-shot learning is a challenging task that requires language models to generalize from limited examples. Large language models like GPT-3 and PaLM have made impressive progress in this area, but they still face difficulties in reasoning tasks such as GSM8K, a benchmark for arithmetic problems. To improve their reasoning skills, previous work has proposed to guide the language model with prompts that elicit a series of reasoning steps before giving the final answer, achieving a significant improvement on GSM8K from $17.9\%$ to $58.1\%$ in problem-solving rate. In this paper, we present DiVeRSe (Diverse Verifier on Reasoning Step), a novel approach that further enhances the reasoning capability of language models. DiVeRSe has three main components: first, it generates diverse prompts to explore different reasoning paths for the same question; second, it uses a verifier to filter out incorrect answers based on a weighted voting scheme; and third, it verifies each reasoning step individually instead of the whole chain. We evaluate DiVeRSe on the latest language model code-davinci-002 and show that it achieves new state-of-the-art results on six of eight reasoning benchmarks (e.g., GS
中文速览
大语言模型在需要多步推理的任务（如算术应用题基准 GSM8K）上仍然表现有限，即便引入了"思维链（chain-of-thought）"提示和多数投票机制，准确率也存在明显天花板。本文提出 DiVeRSe（Diverse Verifier on Reasoning Step）方法，从三个维度同时发力：用多套随机采样的提示词引导模型生成更多样化的推理路径，训练一个验证器（verifier）对每条推理路径打分并加权投票、取代简单多数表决，以及在步骤级别而非整条链级别判断对错，让验证器能精准定位推理在哪一步出了问题。在八个推理基准上的实验表明，DiVeRSe 在六个任务上刷新了最优成绩，例如 GSM8K 从 74.4% 提升到 83.2%，CLUTRR 从 67.0% 跃升至 95.9%，且对不同规模的模型均有稳定增益。这项工作表明，通过多样化探索加细粒度验证的组合，可以在不增大模型参数的前提下大幅释放大模型的推理潜力。
原文 arXiv:2206.02336;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.02336v3