Graph Constrained Reinforcement Learning for Natural Language Action Spaces
Prithviraj Ammanabrolu Georgia Institute of Technology、Matthew Hausknecht Microsoft Research
Abstract
Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C111Code available at https://github.com/rajammanabrolu/KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.
中文速览
文字冒险游戏（Interactive Fiction）要求智能体完全用自然语言与世界交互，但哪怕只有700个词的词表，五词指令就能产生超过1千万亿种可能动作，如何在这种天文数字般的行动空间里高效探索和决策，一直是强化学习面临的核心难题。研究者提出了KG-A2C智能体，核心思路是让智能体一边探索游戏世界，一边动态构建知识图谱来表示当前游戏状态，同时借助"模板+填词"的结构化方式生成自然语言指令，并用知识图谱中的可交互对象进一步限制填词范围，从而将有效行动空间压缩到可管理的规模。在Jericho框架涵盖的多种人工制作文字冒险游戏上，KG-A2C的得分全面超越了此前最优智能体，即便面对指数级更大的行动空间依然保持领先。这项工作的意义在于，它证明了知识图谱"一石二鸟"的价值——既能帮助智能体理解部分可观测的游戏状态，又能约束自然语言生成，为将强化学习扩展到真实复杂的自然语言交互场景提供了一条可行路径。
原文 arXiv:2001.08837;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2001.08837v1