Overcoming Exploration in Reinforcement Learning with Demonstrations
Ashvin Nair12 Bob McGrew1 Marcin Andrychowicz1 Wojciech Zaremba1 Pieter Abbeel12
Abstract
Exploration in environments with sparse rewards has been a persistent problem in reinforcement learning (RL). Many tasks are natural to specify with a sparse reward, and manually shaping a reward function can result in suboptimal performance. However, finding a non-zero reward is exponentially more difficult with increasing task horizon or action dimensionality. This puts many real-world tasks out of practical reach of RL methods. In this work, we use demonstrations to overcome the exploration problem and successfully learn to perform long-horizon, multi-step robotics tasks with continuous control such as stacking blocks with a robot arm. Our method, which builds on top of Deep Deterministic Policy Gradients and Hindsight Experience Replay, provides an order of magnitude of speedup over RL on simulated robotics tasks. It is simple to implement and makes only the additional assumption that we can collect a small set of demonstrations. Furthermore, our method is able to solve tasks not solvable by either RL or behavior cloning alone, and often ends up outperforming the demonstrator policy.
中文速览
在稀疏奖励信号下探索长时程机器人操作任务（如用机械臂叠积木）对强化学习（RL）来说极其困难，因为智能体几乎碰不到任何正向反馈。为此，研究者提出了一种将少量人类演示（100条遥操作示例）融入RL训练全程的方法：在DDPG与事后经验回放（Hindsight Experience Replay, HER）的基础上，额外引入演示缓冲区、行为克隆辅助损失、Q值过滤器（只在演示动作优于当前策略时才施加模仿约束），以及从演示状态直接重置训练回合的机制，从而快速跨越随机探索的瓶颈期。实验结果表明，该方法在仿真机器人任务上比纯RL快约一个数量级，还能解决纯RL和纯行为克隆都无法完成的多步叠块任务，并且最终性能往往超越演示者本身。这项工作的意义在于，它证明了少量演示可以与RL深度结合、贯穿训练始终，而非仅用于初始化，为现实中稀疏奖励、长时程的机器人任务提供了一条切实可行的路径。
原文 arXiv:1709.10089;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.10089v2