Playing Text-Adventure Games with Graph-Based Deep Reinforcement Learning
Prithviraj Ammanabrolu School of Interactive Computing Georgia Institute of Technology Atlanta, GA、Mark O. Riedl School of Interactive Computing Georgia Institute of Technology Atlanta, GA
Abstract
Text-based adventure games provide a platform on which to explore reinforcement learning in the context of a combinatorial action space, such as natural language. We present a deep reinforcement learning architecture that represents the game state as a knowledge graph which is learned during exploration. This graph is used to prune the action space, enabling more efficient exploration. The question of which action to take can be reduced to a question-answering task, a form of transfer learning that pre-trains certain parts of our architecture. In experiments using the TextWorld framework, we show that our proposed technique can learn a control policy faster than baseline alternatives. We have also open-sourced our code at https://github.com/rajammanabrolu/KG-DQN.
中文速览
文字冒险游戏（text-based adventure game）要求智能体在庞大的自然语言动作空间中探索，传统强化学习方法因此收敛极慢。研究者提出了一套名为 KG-DQN 的深度强化学习架构，让智能体在探索过程中自动构建并维护一张知识图谱（knowledge graph）来表示游戏状态，再用图谱中实体与关系的连通性对候选动作打分、剪枝，大幅缩小每一步需要评估的动作数量；同时借助图注意力网络（Graph Attention Network）和双向 LSTM 分别编码图谱与文本观测，并将动作选择类比为问答任务，用已有问答方法预训练网络的部分参数。在 TextWorld 框架上的实验表明，加入知识图谱后收敛速度比最强基线快 40% 以上，再叠加问答预训练则在收敛速度和任务完成质量上同时取得提升。这项工作为在开放语言空间中实现高效强化学习探索提供了一条可行路径，对构建能够通过自然语言理解并改变世界的智能体具有重要参考价值。
原文 arXiv:1812.01628;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1812.01628v2