Learning Dynamic Belief Graphs to Generalize on Text-Based Games
Ashutosh Adhikari Xingdi Yuan Marc-Alexandre Côté Thanks: Equal contribution. Mikuláš Zelinka Marc-Antoine Rondeau Affiliation: University of Waterloo Microsoft Research, Montréal Charles University Romain Laroche Pascal Poupart Jian Tang Adam Trischler William L. Hamilton
Abstract
Playing text-based games requires skills in processing natural language and sequential decision making. Achieving human-level performance on text-based games remains an open challenge, and prior research has largely relied on hand-crafted structured representations and heuristics. In this work, we investigate how an agent can plan and generalize in text-based games using graph-structured representations learned end-to-end from raw text. We propose a novel graph-aided transformer agent (GATA) that infers and updates latent belief graphs during planning to enable effective action selection by capturing the underlying game dynamics. GATA is trained using a combination of reinforcement and self-supervised learning. Our work demonstrates that the learned graph-based representations help agents converge to better policies than their text-only counterparts and facilitate effective generalization across game configurations. Experiments on 500+ unique games from the TextWorld suite show that our best agent outperforms text-based baselines by an average of 24.2%.
中文速览
文字冒险游戏要求智能体从不完整的文本线索中理解环境、记住变化，并在漫长且奖励稀少的过程中做出正确行动，现有方法却常依赖人工规则和结构化信息。研究者提出图辅助 Transformer 智能体（Graph-Aided Transformer Agent，GATA），让模型直接从文本中学习并持续更新“信念图”（belief graph），再结合强化学习和文本重构、对比学习等自监督方法选择行动。 在 TextWorld 生成的500多款不同游戏中，GATA在未见过的游戏配置上比纯文本基线平均高出24.2%，表现接近能够直接读取真实游戏状态的模型。结果说明，自动学出的图结构既能充当部分可观测环境中的记忆，也能帮助智能体理解游戏动态并迁移到新任务，为减少手工设计、构建更通用的语言交互智能体提供了重要方向。
原文 arXiv:2002.09127;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.09127v4