Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models
Kurtland Chua Roberto Calandra Rowan McAllister Sergey Levine Berkeley Artificial Intelligence Research University of California, Berkeley {kchua, roberto.calandra, rmcallister,
Abstract
Model-based reinforcement learning (RL) algorithms can attain excellent sample efficiency, but often lag behind the best model-free algorithms in terms of asymptotic performance. This is especially true with high-capacity parametric function approximators, such as deep networks. In this paper, we study how to bridge this gap, by employing uncertainty-aware dynamics models. We propose a new algorithm called probabilistic ensembles with trajectory sampling (PETS) that combines uncertainty-aware deep network dynamics models with sampling-based uncertainty propagation. Our comparison to state-of-the-art model-based and model-free deep RL algorithms shows that our approach matches the asymptotic performance of model-free algorithms on several challenging benchmark tasks, while requiring significantly fewer samples (e.g., 8 and 125 times fewer samples than Soft Actor Critic and Proximal Policy Optimization respectively on the half-cheetah task).
中文速览
基于模型的强化学习（model-based RL）通常比无模型方法（model-free RL）学得更快，但最终性能往往更差，作者希望填平这一鸿沟。他们提出了一种名为 PETS（Probabilistic Ensembles with Trajectory Sampling，概率集成轨迹采样）的算法，核心做法是把多个"概率神经网络"（每个网络直接输出状态分布而非点预测）组成自举集成（bootstrapped ensemble），同时区分并处理系统固有的随机性（aleatoric uncertainty）和数据不足导致的主观不确定性（epistemic uncertainty），再配合基于采样的不确定性传播来做规划。实验在多个标准控制基准任务上表明，PETS 的渐近性能与顶尖的无模型算法（如 SAC、PPO）相当，所需样本量却大幅减少——在 half-cheetah 任务上比 SAC 少 8 倍、比 PPO 少 125 倍。这项工作说明，只要正确地把不确定性引入深度动力学模型，基于模型的方法完全可以在保持样本高效的同时达到无模型方法的性能上限，为实际机器人等数据昂贵的场景提供了切实可行的路径。
原文 arXiv:1805.12114;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1805.12114v2