A Minimalist Approach to Offline Reinforcement Learning
Scott Fujimoto1,2 Shixiang Shane Gu2 1Mila, McGill University 2Google Research, Brain Team
Abstract
Offline reinforcement learning (RL) defines the task of learning from a fixed batch of data. Due to errors in value estimation from out-of-distribution actions, most offline RL algorithms take the approach of constraining or regularizing the policy with the actions contained in the dataset. Built on pre-existing RL algorithms, modifications to make an RL algorithm work offline comes at the cost of additional complexity. Offline RL algorithms introduce new hyperparameters and often leverage secondary components such as generative models, while adjusting the underlying RL algorithm. In this paper we aim to make a deep RL algorithm work while making minimal changes. We find that we can match the performance of state-of-the-art offline RL algorithms by simply adding a behavior cloning term to the policy update of an online RL algorithm and normalizing the data. The resulting algorithm is a simple to implement and tune baseline, while more than halving the overall run time by removing the additional computational overhead of previous methods.
中文速览
离线强化学习（offline RL）的核心难题是：智能体只能从一批固定数据中学习，一旦它选择了数据集里没有的动作，价值估计就会严重出错，导致策略崩溃。现有解决方案普遍在已有在线RL算法之上堆叠生成模型、额外预训练、网络结构改动等复杂组件，不仅难以复现和调参，运行时间也大幅增加。本文提出的方法TD3+BC极其简洁：在TD3的策略更新目标函数里加一个行为克隆（behavior cloning）正则项，再对状态做零均值单位方差归一化，仅此而已——前后只改动了一行核心代码，引入一个超参数。在D4RL连续控制基准上，TD3+BC的得分与当前最优的离线RL算法持平甚至更好，同时把整体运行时间砍掉一半以上。这项工作的意义在于揭示：离线RL领域长期被复杂算法主导，而足够简单、可解释的基线其实一直被低估，为后续研究提供了一个清晰、可信的参照点。
原文 arXiv:2106.06860;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.06860v2