Reasoning with Language Model is Planning with World Model
Shibo Hao∗♣absent♣*\clubsuit Yi Gu∗♣absent♣*\clubsuit Haodi Ma♢♢\diamondsuit Joshua Jiahua Hong♣♣\clubsuit Zhen Wang♣♣\clubsuit ♠♠\spadesuit Daisy Zhe Wang♢♢\diamondsuit Zhiting Hu♣♣\clubsuit ♣♣\clubsuitUC San Diego, ♢♢\diamondsuitUniversity of Florida ♠♠\spadesuitMohamed bin Zayed University of Artificial Intelligence {s5hao, yig025, jjhong, zhw085, {ma.haodi, equal contribution
Abstract
Large language models (LLMs) have shown remarkable reasoning capabilities, particularly with chain-of-thought (CoT) prompting. However, LLMs sometimes still struggle with problems that are easy for humans, such as generating action plans to achieve given goals in an environment, or performing complex math or logical reasoning. The deficiency stems from the key fact that LLMs lack an internal world model to predict the world state (e.g., environment status, intermediate variable values) and simulate long-term outcomes of actions. This prevents LLMs from performing deliberate planning akin to human brains, which involves exploring alternative reasoning paths, anticipating future states and rewards, and iteratively refining existing reasoning steps. To overcome the limitations, we propose a new LLM reasoning framework, Reasoning via Planning (RAP). RAP repurposes the LLM as both a world model and a reasoning agent, and incorporates a principled planning algorithm based on Monte Carlo Tree Search for strategic exploration in the vast reasoning space. During reasoning, the LLM (as agent) incrementally builds a reasoning tree under the guidance of the LLM (as world model) and rewards, an
中文速览
大语言模型（LLM）在推理任务上仍常常出错，根本原因在于它缺乏一个内部"世界模型"来预测行动后环境会变成什么样，只能一路往前、自回归地生成答案，无法回头探索别的可能性。为此，研究者提出了"规划式推理"框架 RAP（Reasoning via Planning），其核心思路是让同一个 LLM 身兼两职——既作为推理主体生成行动，又作为世界模型预测下一步状态——再结合蒙特卡洛树搜索（Monte Carlo Tree Search, MCTS）算法在庞大的推理空间中有策略地探索，同时用精心设计的奖励函数来评价每一步推理的好坏并反向更新决策。在积木世界规划、数学推理和逻辑推断等多项挑战性任务上，RAP 显著超越了思维链（Chain-of-Thought）等强基线，用 LLaMA-33B 配合 RAP 甚至比 GPT-4 加思维链的方案在规划任务上高出 33%。这项工作表明，把"世界模型＋有计划的搜索"引入 LLM 推理流程，是弥补现有模型在需要前瞻性思考的任务上短板的一条有效路径。
原文 arXiv:2305.14992;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.14992v2