Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems
Sergey Levine1,2, Aviral Kumar1, George Tucker2, Justin Fu1 1UC Berkeley, 2Google Research, Brain Team
Abstract
In this tutorial article, we aim to provide the reader with the conceptual tools needed to get started on research on offline reinforcement learning algorithms: reinforcement learning algorithms that utilize previously collected data, without additional online data collection. Offline reinforcement learning algorithms hold tremendous promise for making it possible to turn large datasets into powerful decision making engines. Effective offline reinforcement learning methods would be able to extract policies with the maximum possible utility out of the available data, thereby allowing automation of a wide range of decision-making domains, from healthcare and education to robotics. However, the limitations of current algorithms make this difficult. We will aim to provide the reader with an understanding of these challenges, particularly in the context of modern deep reinforcement learning methods, and describe some potential solutions that have been explored in recent work to mitigate these challenges, along with recent applications, and a discussion of perspectives on open problems in the field.
中文速览
离线强化学习（offline reinforcement learning）想解决这样一个问题：如何仅凭已有的历史数据，而不需要在真实环境中再收集新数据，就训练出能做出优质决策的智能体。文章系统梳理了该领域的核心挑战——尤其是"分布偏移"（distributional shift）问题，即训练数据的分布与学到的策略实际访问的状态分布不一致，会导致深度神经网络在未见过的状态上给出严重失真的价值估计。针对这一挑战，作者介绍了近年来提出的多类解决思路，包括策略约束、保守价值估计、模型辅助方法等，并综述了离线强化学习在医疗、机器人、对话系统等领域的实际应用。这篇教程文章的意义在于：它为希望入门该领域的研究者提供了系统性的概念框架，同时也点明了当前方法的局限与尚待解决的开放问题，有助于推动离线强化学习走向像监督学习那样"用大数据直接训练强决策引擎"的宏大目标。
原文 arXiv:2005.01643;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.01643v3