Learning Dynamic Belief Graphs to Generalize on Text-Based Games

中文速览

文字冒险游戏要求智能体从不完整的文本线索中理解环境、记住变化，并在漫长且奖励稀少的过程中做出正确行动，现有方法却常依赖人工规则和结构化信息。研究者提出图辅助 Transformer 智能体（Graph-Aided Transformer Agent，GATA），让模型直接从文本中学习并持续更新“信念图”（belief graph），再结合强化学习和文本重构、对比学习等自监督方法选择行动。 在 TextWorld 生成的500多款不同游戏中，GATA在未见过的游戏配置上比纯文本基线平均高出24.2%，表现接近能够直接读取真实游戏状态的模型。结果说明，自动学出的图结构既能充当部分可观测环境中的记忆，也能帮助智能体理解游戏动态并迁移到新任务，为减少手工设计、构建更通用的语言交互智能体提供了重要方向。

摘要

Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%.

术语表

text-based games

文本游戏

sequential decision making

序列决策

natural language processing (NLP)

自然语言处理（NLP）

graph-structured representations

图结构表示

graph-aided transformer agent (GATA)

图辅助 Transformer 智能体（GATA）

latent belief graph

潜在信念图

reinforcement learning (RL)

强化学习（RL）

self-supervised learning

自监督学习

text reconstruction

文本重构

mutual information maximization

互信息最大化

knowledge graph (KG)

知识图谱（KG）

heuristic

启发式方法

partial observability

部分可观测性

partially observable Markov decision process (POMDP)

部分可观测马尔可夫决策过程（POMDP）

sparse reward

稀疏奖励

combinatorial action space

组合式动作空间

inductive bias

归纳偏置

generalization

泛化

TextWorld

TextWorld

dynamic graph

动态图

belief state

信念状态

ground-truth game state

真实游戏状态

graph updater

图更新器

action selector

动作选择器

recurrent policy

循环策略

policy optimization

策略优化

end-to-end learning

端到端学习