Multi-step Off-policy Learning Without Importance Sampling Ratios
\name A. Rupam Mahmood Huizhen Yu Richard S. Sutton \addrReinforcement Learning and Artificial Intelligence Laboratory Department of Computing Science, University of Alberta Edmonton, AB T6G 2E8 Canada
Abstract
To estimate the value functions of policies from exploratory data, most model-free off-policy algorithms rely on importance sampling, where the use of importance sampling ratios often leads to estimates with severe variance. It is thus desirable to learn off-policy without using the ratios. However, such an algorithm does not exist for multi-step learning with function approximation. In this paper, we introduce the first such algorithm based on temporal-difference (TD) learning updates. We show that an explicit use of importance sampling ratios can be eliminated by varying the amount of bootstrapping in TD updates in an action-dependent manner. Our new algorithm achieves stability using a two-timescale gradient-based TD update. A prior algorithm based on lookup table representation called Tree Backup can also be retrieved using action-dependent bootstrapping, becoming a special case of our algorithm. In two challenging off-policy tasks, we demonstrate that our algorithm is stable, effectively avoids the large variance issue, and can perform substantially better than its state-of-the-art counterpart.
中文速览
离线强化学习中，如果想在不实际执行目标策略的情况下评估它的好坏，通常需要用重要性采样（importance sampling）来校正行为策略与目标策略之间的分布差异，但这个校正系数的高度波动会带来极大的估计方差，在多步学习（multi-step learning）场景下问题尤为严重。这篇论文提出了一种名为 ABQ 的新算法，核心思路是用"动作相关的自举参数"（action-dependent bootstrapping parameter）来替代重要性采样比值——通过对不同状态-动作对灵活调整 TD 更新中的自举程度，在数学上等效地消除了对重要性采样比值的显式依赖，并借助双时间尺度梯度更新保证算法稳定性。理论上，ABQ 首次实现了在函数逼近（function approximation）条件下无需重要性采样比值的多步离策略学习，经典的 Tree Backup 算法也作为其特例被统一纳入框架。在两个具有挑战性的离策略任务上，ABQ 表现出比现有最优算法更低的方差和更好的整体性能，为多步离策略强化学习提供了一条实用且理论扎实的新路径。
原文 arXiv:1702.03006;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1702.03006v1