Safe and efficient off-policy reinforcement learning
Rémi Munos Google DeepMind、Thomas Stepleton Google DeepMind \ANDAnna Harutyunyan Vrije Universiteit Brussel、Marc G. Bellemare Google DeepMind
Abstract
In this work, we take a fresh look at some old and new algorithms for off-policy, return-based reinforcement learning. Expressing these in a common form, we derive a novel algorithm, Retrace( $\lambda$ ), with three desired properties: (1) it has low variance; (2) it safely uses samples collected from any behaviour policy, whatever its degree of “off-policyness”; and (3) it is efficient as it makes the best use of samples collected from near on-policy behaviour policies. We analyze the contractive nature of the related operator under both off-policy policy evaluation and control settings and derive online sample-based algorithms. We believe this is the first return-based off-policy control algorithm converging a.s. to $Q^{*}$ without the GLIE assumption (Greedy in the Limit with Infinite Exploration). As a corollary, we prove the convergence of Watkins’ Q( $\lambda$ ), which was an open problem since 1989. We illustrate the benefits of Retrace( $\lambda$ ) on a standard suite of Atari 2600 games.
中文速览
强化学习中有一个核心矛盾：基于完整回报（return）的方法学习效率高，但难以安全地用于离策略（off-policy）数据；而离策略方法又恰恰是经验回放、探索等场景的刚需。Retrace(λ) 算法通过将重要性采样比率截断至1（即取 min(1, π/μ)）来设计迹系数，巧妙地统一了现有几类算法的优点：既像 Tree-backup 一样对任意行为策略保持安全收缩，又像 Q^π(λ) 一样在近似同策略时能充分利用完整回报，同时还避免了重要性采样连乘导致的方差爆炸。理论上，作者证明了 Retrace(λ) 算子是以 γ 为系数的压缩映射，并在控制场景下无需 GLIE 假设即可几乎必然收敛到最优 Q*，还顺带解决了 Watkins' Q(λ) 自1989年以来悬而未决的收敛性证明；在 Atari 2600 游戏基准上的实验也验证了其实际优势。这是首个同时具备安全性、高效性和严格收敛保证的离策略回报学习算法，对深度强化学习中的经验回放等应用具有重要意义。
原文 arXiv:1606.02647;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.02647v2