SmartPlay : A Benchmark for LLMs as Intelligent Agents
Yue Wu12, Xuan Tang1, Tom Mitchell1, Yuanzhi Li12 1Carnegie Mellon University, 2Microsoft Research Work done during internship at Microsoft Research. Correspondence to
Abstract
Recent large language models (LLMs) have demonstrated great potential toward intelligent agents and next-gen automation, but there currently lacks a systematic benchmark for evaluating LLMs’ abilities as agents. We introduce SmartPlay: both a challenging benchmark and a methodology for evaluating LLMs as agents. SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game features a unique setting, providing up to 20 evaluation settings and infinite environment variations. Each game in SmartPlay uniquely challenges a subset of 9 important capabilities of an intelligent LLM agent, including reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness. The distinction between the set of capabilities each game test allows us to analyze each capability separately. SmartPlay serves not only as a rigorous testing ground for evaluating the overall performance of LLM agents but also as a road-map for identifying gaps in current methodologies. We release our benchmark at github.com/microsoft/SmartPlay.
中文速览
大型语言模型（LLM）能否真正胜任"智能体"角色，此前一直缺乏系统性的测试框架。研究者推出了 SmartPlay 这一基准测试，选取了包括石头剪刀布、河内塔、Minecraft 在内的 6 款风格各异的游戏，从长文本理解、多步规划、空间推理、概率判断、从交互中学习等 9 个维度全面考察 LLM 的智能体能力。实验结果表明，当前主流 LLM 在长程规划、空间推理和处理随机性等方面仍存在明显短板，即便是简单的三阶河内塔，模型也难以可靠地记忆中间状态。SmartPlay 不仅提供了一条覆盖多种难度的自动化评测流水线，还为未来研究指明了具体的能力缺口，对推动 LLM 从语言任务走向真实世界自主决策具有重要意义。
原文 arXiv:2310.01557;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.01557v5