PAL: Program-aided Language Models
Luyu Gao Aman Madaan Shuyan Zhou Uri Alon Pengfei Liu Yiming Yang Jamie Callan Graham Neubig
Abstract
Large language models (LLMs) have recently demonstrated an impressive ability to perform arithmetic and symbolic reasoning tasks, when provided with a few examples at test time (“few-shot prompting”). Much of this success can be attributed to prompting methods such as “chain-of-thought”, which employ LLMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LLMs seem to be adept at this sort of step-by-step decomposition, LLMs often make logical and arithmetic mistakes in the solution part, even when the problem is decomposed correctly. In this paper, we present Program-Aided Language models (PaL): a novel approach that uses the LLM to read natural language problems and generate programs as the intermediate reasoning steps, but offloads the solution step to a runtime such as a Python interpreter. With PaL, decomposing the natural language problem into runnable steps remains the only learning task for the LLM, while solving is delegated to the interpreter. We demonstrate this synergy between a neural LLM and a symbolic interpreter across 13 mathematical, symbolic, and algorithmic reasoning tasks from BIG-Benc
中文速览
大语言模型（LLM）在数学和符号推理任务中虽然能把问题拆解成步骤，却经常在具体计算环节出错——哪怕推理思路完全正确，最终答案也可能算错。为此，研究者提出了"程序辅助语言模型"（Program-Aided Language Models，PaL）：让LLM负责读懂题意并生成带注释的Python代码作为推理步骤，把真正的计算和求解工作交给Python解释器来完成，从而把"语言理解"和"精确计算"两件事分给最擅长它的工具去做。在横跨13个数学、符号和算法推理基准的测试中，使用Codex的PaL全面超越了包括参数量大得多的PaLM-540b在内的各类链式思维（chain-of-thought）方法，例如在gsm8k数学应用题基准上绝对准确率提升15%，在包含大数字的难版本gsm-hard上更是领先40%。这项工作表明，将神经语言模型的语言理解能力与符号解释器的精确执行能力相结合，是构建可靠AI推理系统的一条极具潜力的路径。
原文 arXiv:2211.10435;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.10435v2