Stochastic Neural Networks for Hierarchical Reinforcement Learning
Carlos Florensa†, Yan Duan‡†{{}^{\dagger}}{{}^{\ddagger}}, Pieter Abbeel§†‡{{}^{\dagger}}{{}^{\ddagger}}{{}^{\mathsection}} † UC Berkeley, Department of Electrical Engineering and Computer Science § International Computer Science Institute ‡ OpenAI
Abstract
Deep reinforcement learning has achieved many impressive results in recent years. However, tasks with sparse rewards or long horizons continue to pose significant challenges. To tackle these important problems, we propose a general framework that first learns useful skills in a pre-training environment, and then leverages the acquired skills for learning faster in downstream tasks. Our approach brings together some of the strengths of intrinsic motivation and hierarchical methods: the learning of useful skill is guided by a single proxy reward, the design of which requires very minimal domain knowledge about the downstream tasks. Then a high-level policy is trained on top of these skills, providing a significant improvement of the exploration and allowing to tackle sparse rewards in the downstream tasks. To efficiently pre-train a large span of skills, we use Stochastic Neural Networks combined with an information-theoretic regularizer. Our experiments111Code available at: https://github.com/florensacc/snn4hrl show222Videos available at: http://bit.ly/snn4hrl-videos that this combination is effective in learning a wide span of interpretable skills in a sample-efficient way, and can
中文速览
稀疏奖励和超长时间步任务一直是深度强化学习（Deep Reinforcement Learning）的老大难问题，因为智能体很难靠随机探索碰到有意义的奖励信号。本文提出了一个两阶段框架：先在一个简单的预训练环境里，让智能体仅凭一个设计极简的"代理奖励"（proxy reward，比如"跑得越快越好"）学会一大批各具特色的技能；然后在真正的下游任务中，用一个高层策略（high-level policy）来调用这些现成技能，从而把原本漫无目的的探索变成有目的的技能组合，大幅降低对稀疏奖励的依赖。技能的多样性靠随机神经网络（Stochastic Neural Network, SNN）加上互信息（Mutual Information）正则项来保证，让不同隐变量自然对应不同的运动模式，比如向不同方向移动或抓取不同物体。实验表明，这套方法能以较高的样本效率学出大量可解释的技能，并在一系列长时间步、稀疏奖励的任务上显著提升学习速度，意味着只需极少的领域知识就能让机器人"先练基本功、再做具体任务"，具有广泛的实用价值。
原文 arXiv:1704.03012;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1704.03012v1