Aha.
正在载入中英对照阅读…

arXiv:2006.15134 · 中英对照阅读

评论家正则化回归

Critic Regularized Regression

Ziyu Wang、Alexander Novikov、Konrad Żołna、Jost Tobias Springenberg、Scott Reed、Bobak Shahriari、Noah Siegel、Josh Merel、Caglar Gulcehre、Nicolas Heess、Nando de Freitas

中文速览

离线强化学习想只靠历史数据学出好策略,难点是策略可能选出数据里没出现过的动作,导致价值估计不准、越学越偏。作者提出“评论家正则化回归”(CRR):先用价值模型评估数据中的动作,再让策略更常模仿价值较高的动作,从而既利用奖励改进策略,又不轻易跑出数据范围。实验显示,CRR在多种基准任务上明显胜过若干先进方法,也能处理状态和动作维度很高、数据质量参差不齐的任务。它让机器人、医疗和工业控制等不便在线试错的场景,更有希望直接从已有记录中学出可靠策略。

摘要

离线强化学习(RL),又称批量强化学习,能够基于大型预先记录的数据集进行策略优化,而无需与在线环境交互。这一方法应对了数据收集成本和安全性方面的挑战,而这两者对于强化学习在现实世界中的应用尤为重要。遗憾的是,大多数离策略算法在固定数据集上学习时表现不佳。本文提出了一种新颖的离线强化学习算法,通过一种评论家正则化回归(CRR)形式,从数据中学习策略。我们发现,CRR 的表现出乎意料地出色,并且能够扩展到具有高维状态空间和动作空间的任务;在广泛的基准任务中,其表现显著优于多种最先进的离线强化学习算法。

Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces – outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.

术语表

Offline reinforcement learning (offline RL)
离线强化学习
Batch reinforcement learning (batch RL)
批量强化学习
Online reinforcement learning (online RL)
在线强化学习
Policy optimization
策略优化
Off-policy algorithm
离策略算法
Behavior cloning (BC)
行为克隆
Function approximation
函数逼近
State space
状态空间
Action space
动作空间
Critic-regularized regression (CRR)
评论家正则化回归
Value-filtered regression
价值过滤回归
Actor-critic method
演员-评论家方法
Q-function
Q函数
Q-estimate
Q值估计
Bootstrapping
自举
Extrapolation error
外推误差
Distributional reinforcement learning
分布式强化学习
Batch-Constrained deep Q-learning (BCQ)
批量约束深度Q学习
Advantage function
优势函数
Monotonic advantage re-weighted imitation learning (MARWIL)
单调优势重加权模仿学习
Best-action imitation learning (BAIL)
最佳动作模仿学习
Advantage-weighted behavior models (ABM)
优势加权行为模型
Advantage-weighted regression
优势加权回归
Critic Weighted Policy (CWP)
评论家加权策略
Advantage Weighted Actor Critic (AWAR)
优势加权演员-评论家
Policy evaluation
策略评估
Temporal difference loss
时序差分损失
Target network
目标网络
Distributional Q-function
分布式Q函数
Fitted Q-iteration
拟合Q迭代