D4RL: Datasets for Deep Data-Driven Reinforcement Learning
Justin Fu UC Berkeley、Aviral Kumar UC Berkeley \ANDOfir Nachum Google Brain、George Tucker Google Brain、Sergey Levine UC Berkeley, Google Brain
Abstract
The offline reinforcement learning (RL) setting (also known as full batch RL), where a policy is learned from a static dataset, is compelling as progress enables RL methods to take advantage of large, previously-collected datasets, much like how the rise of large datasets has fueled results in supervised learning. However, existing online RL benchmarks are not tailored towards the offline setting and existing offline RL benchmarks are restricted to data generated by partially-trained agents, making progress in offline RL difficult to measure. In this work, we introduce benchmarks specifically designed for the offline setting, guided by key properties of datasets relevant to real-world applications of offline RL. With a focus on dataset collection, examples of such properties include: datasets generated via hand-designed controllers and human demonstrators, multitask datasets where an agent performs different tasks in the same environment, and datasets collected with mixtures of policies. By moving beyond simple benchmark tasks and data collected by partially-trained RL agents, we reveal important and unappreciated deficiencies of existing algorithms. To facilitate research, we have
中文速览
离线强化学习（offline RL）希望像监督学习利用大规模数据集那样，直接从已有数据中训练出优秀策略，却一直缺乏能真实反映实际应用场景的标准测试基准。为此，论文推出了 D4RL 基准套件，涵盖迷宫导航、机器人控制、自动驾驶等多类任务，并专门收集了人类示范、手工控制器生成、多任务混合等多种来源的数据集，而非像以往那样只用在线 RL 训练过程中产生的数据。用这套基准对当前主流离线 RL 算法进行全面评测后发现，大多数算法在数据分布狭窄、被动采集或来自人类示范的场景下表现明显不足，暴露出此前简单基准所掩盖的关键缺陷。D4RL 随论文一并开源，为整个社区提供了统一的起点，有助于推动离线强化学习走向真实可用的应用场景。
原文 arXiv:2004.07219;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.07219v4