SPRING: Studying the Paper and Reasoning to Play Games
Yue Wu14, Shrimai Prabhumoye2, So Yeon Min1, Yonatan Bisk1, Ruslan Salakhutdinov1, Amos Azaria3, Tom Mitchell1, Yuanzhi Li1,4 1Carnegie Mellon University, 2NVIDIA, 3Ariel University, 4Microsoft Research Work done during internship at Microsoft. For correspondence, contact
Abstract
Open-world survival games pose significant challenges for AI algorithms due to their multi-tasking, deep exploration, and goal prioritization requirements. Despite reinforcement learning (RL) being popular for solving games, its high sample complexity limits its effectiveness in complex open-world games like Crafter or Minecraft. We propose a novel approach, SPRING, to read Crafter’s original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM). Prompted with the LaTeX source as game context and a description of the agent’s current observation, our SPRING framework employs a directed acyclic graph (DAG) with game-related questions as nodes and dependencies as edges. We identify the optimal action to take in the environment by traversing the DAG and calculating LLM responses for each node in topological order, with the LLM’s answer to final node directly translating to environment actions. In our experiments, we study the quality of in-context “reasoning” induced by different forms of prompts under the setting of the Crafter environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have gr
中文速览
开放世界生存游戏（如 Crafter）要求 AI 同时处理多任务、规划科技树、协调冲突目标，传统强化学习（RL）因样本效率低、难以融入先验知识而表现有限。SPRING 框架另辟蹊径：先让大语言模型（LLM）阅读 Crafter 原始学术论文的 LaTeX 源码，从中提取游戏机制与交互规则，再通过一个以问题为节点、依赖关系为有向边的有向无环图（DAG）引导 LLM 逐步推理，最终将回答直接映射为游戏动作。实验表明，用 GPT-4 驱动的 SPRING 在零样本（无需任何训练）条件下，游戏得分比经过 100 万步训练的最优 RL 基线高出约 88%。这一结果说明，将结构化推理与文献知识相结合，可以极大弥补 RL 在复杂开放世界任务中的不足，也为用游戏环境评测 LLM 的规划与推理能力提供了新思路。
原文 arXiv:2305.15486;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.15486v3