Rational Decision-Making Agent with Learning Internal Utility Judgment
Yining Ye1 , Xin Cong1∗ , Shizuo Tian1, Yujia Qin1, Chong Liu1, Yankai Lin†2{}^{2}{\dagger}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPT †, Zhiyuan Liu1, Maosong Sun1 1Tsinghua University 2Renmin University of China Indicates equal contribution. Corresponding author.
Abstract
With remarkable advancements, large language models (LLMs) have attracted significant efforts to develop LLM-based agents capable of executing intricate multi-step decision-making tasks. Existing approaches predominantly build upon the external performance measure to guide the decision-making process but the reliance on the external performance measure as prior is problematic in real-world scenarios, where such prior may be unavailable, flawed, or even erroneous. For genuine autonomous decision-making for LLM-based agents, it is imperative to develop rationality from their posterior experiences to judge the utility of each decision independently. In this work, we propose RaDAgent (Rational Decision-Making Agent), which fosters the development of its rationality through an iterative framework involving Experience Exploration and Utility Learning. Within this framework, Elo-based Utility Learning is devised to assign Elo scores to individual decision steps to judge their utilities via pairwise comparisons. Consequently, these Elo scores guide the decision-making process to derive optimal outcomes. Experimental results on the Game of 24, WebShop, ToolBench and RestBench datasets demon
中文速览
现有的大语言模型（LLM）智能体在做多步决策时，普遍依赖人工设计的外部评估指标来引导决策，但这些指标在真实场景中往往缺失、不准确甚至是错误的。为解决这一问题，研究者提出了 RaDAgent（理性决策智能体），让模型通过自身的探索经验来学习每一步决策的内在效用，而不依赖任何外部先验。具体做法是：先用类似蒙特卡洛树搜索（Monte Carlo Tree Search）的方式广泛探索可能的决策路径，再借助竞技评级中常用的 Elo 评分系统，通过对不同决策路径进行两两比较来为每个决策步骤打分，分数高的路径反过来又指导下一轮探索，如此迭代直到找到最优解。在 Game of 24、WebShop、ToolBench 和 RestBench 四个基准数据集上的实验表明，RaDAgent 平均比现有方法提升约 7.8%，同时还减少了 ChatGPT API 的调用次数，在效果和效率上均有显著优势。这项工作的意义在于，它为 LLM 智能体提供了一条无需依赖外部反馈就能自主形成理性判断的新路径，对真实世界中复杂任务的自主决策具有重要的实用价值。
原文 arXiv:2308.12519;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.12519v3