RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning
Caglar GulcehreD,D{}^{\text{D},} Ziyu WangG,∗G∗{}^{\text{G},\ast} Alexander NovikovD,∗D∗{}^{\text{D},\ast} Tom Le PaineD,∗D∗{}^{\text{D},\ast} Sergio Gómez ColmenarejoDD{}^{\text{D}} Konrad ŻołnaDD{}^{\text{D}} Rishabh AgarwalGG{}^{\text{G}} Josh MerelDD{}^{\text{D}} Daniel MankowitzDD{}^{\text{D}} Cosmin PaduraruDD{}^{\text{D}} Gabriel Dulac-ArnoldGG{}^{\text{G}} Jerry LiDD{}^{\text{D}} Mohammad NorouziGG{}^{\text{G}} Matt HoffmanDD{}^{\text{D}} Nicolas HeessDD{}^{\text{D}} Nando de FreitasDD{}^{\text{D}} D: DeepMind G: Google Brain Indicates joint first authors.
Abstract
Offline methods for reinforcement learning have a potential to help bridge the gap between reinforcement learning research and real-world applications. They make it possible to learn policies from offline datasets, thus overcoming concerns associated with online data collection in the real-world, including cost, safety, or ethical concerns. In this paper, we propose a benchmark called RL Unplugged to evaluate and compare offline RL methods. RL Unplugged includes data from a diverse range of domains including games (e.g., Atari benchmark) and simulated motor control problems (e.g., DM Control Suite). The datasets include domains that are partially or fully observable, use continuous or discrete actions, and have stochastic vs. deterministic dynamics. We propose detailed evaluation protocols for each domain in RL Unplugged and provide an extensive analysis of supervised learning and offline RL methods using these protocols. We will release data for all our tasks and open-source all algorithms presented in this paper. We hope that our suite of benchmarks will increase the reproducibility of experiments and make it possible to study challenging tasks with a limited computational budget
中文速览
强化学习在现实世界中落地难，很大原因是在线数据采集成本高、风险大，而离线强化学习（Offline RL）恰好能从已有的历史数据中学习策略、绕开这一障碍，但该领域长期缺乏统一的数据集、评测协议和可复现的基线，导致不同算法之间很难公平比较。为此，作者推出了名为"RL Unplugged"的离线RL综合基准，收录了Atari游戏、DM Control Suite机器人运动控制、DM Locomotion、真实世界RL挑战（RWRL）等多个领域的大规模离线数据集，覆盖离散/连续动作、高维图像/低维状态观测、部分可观测性、随机动力学等多种难度维度。他们同时提出了在线策略选择和离线策略选择两套评估协议，并对多种监督学习与最新离线RL算法进行了系统性对比实验，全部数据和代码均开源发布。这项工作的重要性在于：它为离线RL研究提供了可复现、可扩展的公共平台，有助于降低计算门槛、推动该领域从学术实验室走向真实应用。
原文 arXiv:2006.13888;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.13888v4