TreeQN and ATreeC: Differentiable Tree-Structured Models for Deep Reinforcement Learning
Gregory Farquhar†、Tim Rocktäschel† \ANDMaximilian Igl†、Shimon Whiteson† \AND†University of Oxford, United Kingdom
Abstract
Combining deep model-free reinforcement learning with on-line planning is a promising approach to building on the successes of deep RL. On-line planning with look-ahead trees has proven successful in environments where transition models are known a priori. However, in complex environments where transition models need to be learned from data, the deficiencies of learned models have limited their utility for planning. To address these challenges, we propose TreeQN, a differentiable, recursive, tree-structured model that serves as a drop-in replacement for any value function network in deep RL with discrete actions. TreeQN dynamically constructs a tree by recursively applying a transition model in a learned abstract state space and then aggregating predicted rewards and state-values using a tree backup to estimate $Q$ -values. We also propose ATreeC, an actor-critic variant that augments TreeQN with a softmax layer to form a stochastic policy network. Both approaches are trained end-to-end, such that the learned model is optimised for its actual use in the tree. We show that TreeQN and ATreeC outperform $n$ -step DQN and A2C on a box-pushing task, as well as $n$ -step DQN and value pr
中文速览
深度强化学习（deep RL）在复杂环境中往往缺乏对未来的规划能力，而已有的基于模型的规划方法又因为学到的环境模型误差累积过快而难以实用。本文提出了 TreeQN 和 ATreeC 两种新架构：在编码当前状态后，通过递归地在抽象潜在空间中展开动作树、预测奖励并用树备份（tree backup）聚合值估计，直接将"向前看"的规划过程嵌入 Q 函数或策略网络本身，整个系统端到端训练，使过渡模型从一开始就为规划目标而优化。实验表明，TreeQN 和 ATreeC 在推箱子任务和多款 Atari 游戏上均优于 n-step DQN、A2C 以及价值预测网络（VPN），更深的树往往带来更好的性能。这项工作的意义在于，它提供了一种将规划结构作为归纳偏置内嵌于神经网络的新思路，让智能体无需精确的环境模拟器便能从有限经验中学到更优的策略。
原文 arXiv:1710.11417;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1710.11417v2