FeUdal Networks for Hierarchical Reinforcement Learning
Alexander Sasha Vezhnevets Simon Osindero Tom Schaul Nicolas Heess Max Jaderberg David Silver Koray Kavukcuoglu DeepMind
Abstract
We introduce FeUdal Networks (FuNs): a novel architecture for hierarchical reinforcement learning. Our approach is inspired by the feudal reinforcement learning proposal of Dayan and Hinton, and gains power and efficacy by decoupling end-to-end learning across multiple levels – allowing it to utilise different resolutions of time. Our framework employs a Manager module and a Worker module. The Manager operates at a lower temporal resolution and sets abstract goals which are conveyed to and enacted by the Worker. The Worker generates primitive actions at every tick of the environment. The decoupled structure of FuN conveys several benefits – in addition to facilitating very long timescale credit assignment it also encourages the emergence of sub-policies associated with different goals set by the Manager. These properties allow FuN to dramatically outperform a strong baseline agent on tasks that involve long-term credit assignment or memorisation. We demonstrate the performance of our proposed system on a range of tasks from the ATARI suite and also from a 3D DeepMind Lab environment.
中文速览
分层强化学习长期以来难以解决稀疏奖励和长程信用分配问题，FeUdal Networks（FuN）通过引入"封建"式的管理者-工人（Manager-Worker）双层架构来应对这一挑战：管理者在较低的时间分辨率下在自学习的隐空间中生成方向性目标，工人则在每一步环境交互中根据这些目标产生具体动作，两者之间不传递梯度，管理者仅靠环境奖励学习、工人靠内在奖励跟随目标。为了更好地训练管理者，研究者提出了"过渡策略梯度"（transition policy gradient）这一新型更新规则，利用余弦相似度衡量状态转移方向与目标方向的对齐程度，使目标具有可解释的语义含义，同时还设计了扩张LSTM（dilated LSTM）使记忆能跨越数百步时间回传梯度。在多款Atari游戏（包括以难著称的Montezuma's Revenge）以及3D的DeepMind Lab记忆任务上，FuN均大幅超越强基线，验证了该架构在长程信用分配和记忆场景下的显著优势，对推动分层强化学习的实用化具有重要意义。
原文 arXiv:1703.01161;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.01161v2