Continuous Deep Q-Learning with Model-based Acceleration
Shixiang Gu1 2 3 Timothy Lillicrap4 Ilya Sutskever3 Sergey Levine3 1University of Cambridge 2Max Planck Institute for Intelligent Systems 3Google Brain 4Google DeepMind
Abstract
Model-free reinforcement learning has been successfully applied to a range of challenging problems, and has recently been extended to handle large neural network policies and value functions. However, the sample complexity of model-free algorithms, particularly when using high-dimensional function approximators, tends to limit their applicability to physical systems. In this paper, we explore algorithms and representations to reduce the sample complexity of deep reinforcement learning for continuous control tasks. We propose two complementary techniques for improving the efficiency of such algorithms. First, we derive a continuous variant of the Q-learning algorithm, which we call normalized adantage functions (NAF), as an alternative to the more commonly used policy gradient and actor-critic methods. NAF representation allows us to apply Q-learning with experience replay to continuous tasks, and substantially improves performance on a set of simulated robotic control tasks. To further improve the efficiency of our approach, we explore the use of learned models for accelerating model-free reinforcement learning. We show that iteratively refitted local linear models are especially e
中文速览
无模型强化学习（model-free RL）虽然已能处理复杂任务，但在真实物理系统上因样本效率太低而难以落地，尤其是配合深度神经网络使用时问题更突出。为此，论文提出两项互补技术：一是设计了一种名为"归一化优势函数"（Normalized Advantage Functions，NAF）的连续动作Q学习方法，将Q函数参数化为关于动作的二次型，从而无需像演员-评论家（actor-critic）方法那样训练两个独立网络，便可直接在连续动作空间中应用经验回放的Q学习；二是将迭代拟合的局部线性动力学模型与"想象展开"（imagination rollouts，即Dyna-Q风格的模型内模拟）相结合，用少量真实数据快速提升样本效率，同时规避全局神经网络动力学模型精度不足带来的误差放大问题。在一系列模拟机器人控制任务上，NAF单独使用就显著优于DDPG等基准方法，结合局部线性模型后学习速度进一步大幅提升。这项工作为把深度强化学习真正用于样本稀缺的真实机器人场景提供了一条切实可行的路径。
原文 arXiv:1603.00748;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.00748v1