\papertitle
Victor Zhong. Paul G. Allen School of Computer Science、Engineering University of Washington、Tim Rocktäschel Facebook AI Research、University College London、Edward Grefenstette Facebook AI Research、University College London Work done during an internship at Facebook AI Research. We open-sourced this project at https://github.com/facebookresearch/RTFM
Abstract
Obtaining policies that can generalise to new environments in reinforcement learning is challenging. In this work, we demonstrate that language understanding via a reading policy learner is a promising vehicle for generalisation to new environments. We propose a grounded policy learning problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason over a language goal, relevant dynamics described in a document, and environment observations. We procedurally generate environment dynamics and corresponding language descriptions of the dynamics, such that agents must read to understand new environment dynamics instead of memorising any particular information. In addition, we propose txt2 $\pi$ , a model that captures three-way interactions between the goal, document, and observations. On RTFM, txt2 $\pi$ generalises to new environments with dynamics not seen during training via reading. Furthermore, our model outperforms baselines such as FiLM and language-conditioned CNNs on RTFM. Through curriculum learning, txt2 $\pi$ produces policies that excel on complex RTFM tasks requiring several reasoning and coreference steps.
中文速览
强化学习（RL）中的智能体通常死记硬背特定环境规则，一旦碰到没见过的新环境便束手无策；为了解决这个泛化难题，研究者提出让智能体像人一样"读说明书"来理解规则，而不是靠记忆。论文设计了一个名为 RTFM（Read to Fight Monsters）的任务环境：智能体每局面对随机生成的怪物、道具和克制关系，必须同时理解文字目标、描述当前规则的文档以及游戏画面，才能找到正确武器并击败指定怪物。针对这一三方联合推理需求，作者提出了 txt2π 模型，其核心是双向特征调制层（FiLM²），能让语言与视觉观察互相过滤、相互增强，从而抽取与当前目标真正相关的信息。实验表明，txt2π 在从未见过的新环境动态上依然保持高胜率，优于 FiLM 等基线模型，经过课程学习后还能处理需要五步跨文档推理的复杂关卡——这说明"读文档来泛化"是让 RL 智能体适应真实世界动态变化的一条可行路径。
原文 arXiv:1910.08210;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.08210v6