arXiv:2006.15134 · 中英对照阅读
评论家正则化回归
Critic Regularized Regression
中文速览
离线强化学习想只靠历史数据学出好策略,难点是策略可能选出数据里没出现过的动作,导致价值估计不准、越学越偏。作者提出“评论家正则化回归”(CRR):先用价值模型评估数据中的动作,再让策略更常模仿价值较高的动作,从而既利用奖励改进策略,又不轻易跑出数据范围。实验显示,CRR在多种基准任务上明显胜过若干先进方法,也能处理状态和动作维度很高、数据质量参差不齐的任务。它让机器人、医疗和工业控制等不便在线试错的场景,更有希望直接从已有记录中学出可靠策略。
摘要
离线强化学习(RL),又称批量强化学习,能够基于大型预先记录的数据集进行策略优化,而无需与在线环境交互。这一方法应对了数据收集成本和安全性方面的挑战,而这两者对于强化学习在现实世界中的应用尤为重要。遗憾的是,大多数离策略算法在固定数据集上学习时表现不佳。本文提出了一种新颖的离线强化学习算法,通过一种评论家正则化回归(CRR)形式,从数据中学习策略。我们发现,CRR 的表现出乎意料地出色,并且能够扩展到具有高维状态空间和动作空间的任务;在广泛的基准任务中,其表现显著优于多种最先进的离线强化学习算法。
Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces – outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.
术语表
- Offline reinforcement learning (offline RL)
- 离线强化学习
- Batch reinforcement learning (batch RL)
- 批量强化学习
- Online reinforcement learning (online RL)
- 在线强化学习
- Policy optimization
- 策略优化
- Off-policy algorithm
- 离策略算法
- Behavior cloning (BC)
- 行为克隆
- Function approximation
- 函数逼近
- State space
- 状态空间
- Action space
- 动作空间
- Critic-regularized regression (CRR)
- 评论家正则化回归
- Value-filtered regression
- 价值过滤回归
- Actor-critic method
- 演员-评论家方法
- Q-function
- Q函数
- Q-estimate
- Q值估计
- Bootstrapping
- 自举
- Extrapolation error
- 外推误差
- Distributional reinforcement learning
- 分布式强化学习
- Batch-Constrained deep Q-learning (BCQ)
- 批量约束深度Q学习
- Advantage function
- 优势函数
- Monotonic advantage re-weighted imitation learning (MARWIL)
- 单调优势重加权模仿学习
- Best-action imitation learning (BAIL)
- 最佳动作模仿学习
- Advantage-weighted behavior models (ABM)
- 优势加权行为模型
- Advantage-weighted regression
- 优势加权回归
- Critic Weighted Policy (CWP)
- 评论家加权策略
- Advantage Weighted Actor Critic (AWAR)
- 优势加权演员-评论家
- Policy evaluation
- 策略评估
- Temporal difference loss
- 时序差分损失
- Target network
- 目标网络
- Distributional Q-function
- 分布式Q函数
- Fitted Q-iteration
- 拟合Q迭代