Variational Intrinsic Control
Karol Gregor, Danilo Rezende and Daan Wierstra DeepMind
Abstract
In this paper we introduce a new unsupervised reinforcement learning method for discovering the set of intrinsic options available to an agent. This set is learned by maximizing the number of different states an agent can reliably reach, as measured by the mutual information between the set of options and option termination states. To this end, we instantiate two policy gradient based algorithms, one that creates an explicit embedding space of options and one that represents options implicitly. The algorithms also provide an explicit measure of empowerment in a given state that can be used by an empowerment maximizing agent. The algorithm scales well with function approximation and we demonstrate the applicability of the algorithm on a range of tasks.
中文速览
智能体在环境里如何自主发现"自己能做哪些有意义的事"，是强化学习长期悬而未决的难题。这项工作提出了一种无监督强化学习方法，用选项（options，即带终止条件的策略）和终止状态之间的互信息（mutual information）作为学习目标，驱动智能体主动探索并掌握尽可能多样的可达状态，从而自动找出它在环境中真正拥有的内在选项集合。与以往只考虑开环动作序列的做法不同，这里训练的是闭环策略——智能体每步都能根据当前状态调整行动，并同时学习一个变分推断网络来辨别"到底是哪个选项把我带到了这里"，两者协同优化使互信息估计更准确、实际可控状态数量更多。实验结果表明，该算法在多种任务上显著优于开环基线，既能为追求外部奖励的智能体提供有用的初始技能表示，也能直接服务于以"赋权度（empowerment）最大化"为目标的内在动机智能体，为无监督控制领域提供了一把类似最大似然之于无监督表示学习的标准量尺。
原文 arXiv:1611.07507;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.07507v1