Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learning
Cameron Voloshin Caltech、Hoang M. Le Argo AI、Nan Jiang UIUC、Yisong Yue Caltech
Abstract
We offer an experimental benchmark and empirical study for off-policy policy evaluation (OPE) in reinforcement learning, which is a key problem in many safety critical applications. Given the increasing interest in deploying learning-based methods, there has been a flurry of recent proposals for OPE method, leading to a need for standardized empirical analyses. Our work takes a strong focus on diversity of experimental design to enable stress testing of OPE methods. We provide a comprehensive benchmarking suite to study the interplay of different attributes on method performance. We also distill the results into a summarized set of guidelines for OPE in practice. Our software package, the Caltech OPE Benchmarking Suite (COBS), is open-sourced and we invite interested researchers to further contribute to the benchmark.
中文速览
离线策略评估（Off-Policy Evaluation, OPE）是强化学习落地安全关键场景的核心问题——它要用历史数据估算一个新策略的表现，而无需真正执行该策略——但现有研究缺乏统一的测试标准，导致不同方法之间的比较难以可信。为此，研究者构建了加州理工OPE基准套件（COBS），通过精心设计的八个实验环境，系统覆盖了影响OPE性能的关键因素，包括时间跨度、奖励稀疏性、环境随机性、策略分布偏移程度以及模型误设定等。他们在这套基准上对多种主流OPE方法进行了全面测试，发现没有哪种方法能在所有条件下稳定领先，方法性能对环境随机性和策略偏移程度极为敏感，并从中提炼出可供从业者参考的实践指南。这项工作填补了OPE领域缺乏可复现、细粒度实验对比平台的空白，开源代码已被多个研究团队采用，有望推动该领域更规范、更深入的方法比较与发展。
原文 arXiv:1911.06854;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.06854v3