Critic Regularized Regression
Ziyu Wang、Alexander Novikov、Konrad Żołna、Jost Tobias Springenberg、Scott Reed、Bobak Shahriari、Noah Siegel、Josh Merel、Caglar Gulcehre、Nicolas Heess、Nando de Freitas DeepMind, London, United Kingdom.Google Brain, Toronto, Canada.
Abstract
Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces – outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.
中文速览
离线强化学习（offline RL）希望只用历史数据就能训练出好策略，而不必让智能体在真实环境中反复试错，这对医疗、工业控制等高风险场景尤为关键。然而直接把常规的离线策略算法用到固定数据集上往往失败，根源在于Q函数会对数据集之外的动作给出过于乐观的估计。本文提出"评论家正则化回归"（Critic-Regularized Regression，CRR）：核心思路是用Q函数（或优势函数）对数据集中的动作打分并加权，只让策略去模仿那些被评价为"值得做"的动作，从而把离线策略优化转化为一种经过质量筛选的行为克隆，几乎不需要改动标准的演员-评论家框架。实验表明，CRR在多种基准任务上显著超越多个现有最优离线RL算法，尤其在高维状态和动作空间以及数据质量参差不齐的情况下仍然表现出色。这项工作证明了一个设计简洁、易于实现的方法完全可以在离线RL这一实用性极强的领域取得领先效果，为真实世界应用提供了可靠的算法基础。
原文 arXiv:2006.15134;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.15134v3