Keep Doing What Worked: Behavior Modelling Priors for Offline Reinforcement Learning
Noah Y. Siegel Jost Tobias Springenberg Felix Berkenkamp Abbas Abdolmaleki Michael Neunert Thomas Lampe Roland Hafner Nicolas Heess Martin Riedmiller DeepMind
Abstract
Off-policy reinforcement learning algorithms promise to be applicable in settings where only a fixed data-set (batch) of environment interactions is available and no new experience can be acquired. This property makes these algorithms appealing for real world problems such as robot control. In practice, however, standard off-policy algorithms fail in the batch setting for continuous control. In this paper, we propose a simple solution to this problem. It admits the use of data generated by arbitrary behavior policies and uses a learned prior – the advantage-weighted behavior model (ABM) – to bias the RL policy towards actions that have previously been executed and are likely to be successful on the new task. Our method can be seen as an extension of recent work on batch-RL that enables stable learning from conflicting data-sources. We find improvements on competitive baselines in a variety of RL tasks – including standard continuous control benchmarks and multi-task learning for simulated and real-world robots. Videos are available at https://sites.google.com/view/behavior-modelling-priors.
中文速览
强化学习通常需要与环境不断交互来收集数据，但在机器人控制等真实场景中，数据采集成本极高，因此"离线批量强化学习"——仅用已有固定数据集来训练策略——变得非常重要。然而直接把现有的离策略（off-policy）算法用在这种纯批量数据上效果很差，尤其当数据来自多个目标不同、甚至相互冲突的行为策略时，算法容易被无关的行为模式带偏。本文提出了一种叫做"优势加权行为模型"（Advantage-Weighted Behavior Model，ABM）的学习先验（prior）：它先对数据集中的行为建模，再用类似"优势函数"的权重把那些对当前任务有益的动作筛选出来，让策略在策略迭代过程中始终贴近这些"既出现在数据里、又对任务有帮助"的动作，从而避免策略跑到数据从未覆盖的危险区域。实验结果表明，该方法在标准连续控制基准、多任务仿真以及真实机器人学习任务上均优于已有的竞争方法，且算法比前人工作更简洁，意味着只要先验设计得当，策略评估步骤会被隐式稳定，无需额外复杂的修正机制——这为在数据有限的真实世界场景中部署强化学习提供了一条更实用的路径。
原文 arXiv:2002.08396;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.08396v3