Behavior Regularized Offline Reinforcement Learning
Yifan Wu Carnegie Mellon University、George Tucker Google Research、Ofir Nachum Google Research Work performed while an intern at Google Brain.
Abstract
In reinforcement learning (RL) research, it is common to assume access to direct online interactions with the environment. However in many real-world applications, access to the environment is limited to a fixed offline dataset of logged experience. In such settings, standard RL algorithms have been shown to diverge or otherwise yield poor performance. Accordingly, recent work has suggested a number of remedies to these issues. In this work, we introduce a general framework, behavior regularized actor critic (BRAC), to empirically evaluate recently proposed methods as well as a number of simple baselines across a variety of offline continuous control tasks. Surprisingly, we find that many of the technical complexities introduced in recent methods are unnecessary to achieve strong performance. Additional ablations provide insights into which design choices matter most in the offline RL setting.111Code is available at https://github.com/google-research/google-research/tree/master/behavior_regularized_offline_rl.
中文速览
离线强化学习（offline RL）面临一个核心难题：智能体只能从固定的历史数据集中学习策略，无法与环境实时交互，导致主流强化学习算法因Q值函数的错误泛化而严重失效。针对这一问题，研究者们近年来提出了多种补救方案，核心思路都是在学习过程中把新策略"拉"向产生数据的行为策略（behavior policy），以避免对未见过的状态-动作对产生过度乐观的估计。本文提出了一个统一算法框架"行为正则化演员-评论家"（Behavior Regularized Actor Critic，BRAC），将BCQ、BEAR等现有方法纳入同一体系，并在多个连续控制任务上系统性地对比了不同设计选择——包括正则化方式（值函数惩罚 vs. 策略惩罚）、散度度量（KL散度、MMD、Wasserstein距离）以及自适应正则化权重等。实验结果出人意料：那些看似精妙的技术细节，如加权Q值集成和自适应正则化权重，并非取得好性能的必要条件，结构简单的基线变体同样能达到甚至超越复杂方法的水平。这一发现提醒研究者重新审视离线强化学习算法设计中哪些是真正关键的因素，避免将偶然的工程迭代误当成不可或缺的算法创新。
原文 arXiv:1911.11361;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.11361v1