Minimax Value Interval for Off-Policy Evaluation and Policy Optimization
Nan Jiang Department of Computer Science University of Illinois at Urbana-Champaign Urbana, IL 61801 Jiawei Huang Department of Computer Science University of Illinois at Urbana-Champaign Urbana, IL 61801
Abstract
We study minimax methods for off-policy evaluation (OPE) using value functions and marginalized importance weights. Despite that they hold promises of overcoming the exponential variance in traditional importance sampling, several key problems remain: (1) They require function approximation and are generally biased. For the sake of trustworthy OPE, is there anyway to quantify the biases? (2) They are split into two styles (“weight-learning” vs “value-learning”). Can we unify them? In this paper we answer both questions positively. By slightly altering the derivation of previous methods (one from each style [1]), we unify them into a single value interval that comes with a special type of double robustness: when either the value-function or the importance-weight class is well specified, the interval is valid and its length quantifies the misspecification of the other class. Our interval also provides a unified view of and new insights to some recent methods, and we further explore the implications of our results on exploration and exploitation in off-policy policy optimization with insufficient data coverage.
中文速览
离线策略评估（off-policy evaluation，OPE）想解决的是：如何用历史数据可靠地估算一个从未真正部署过的新策略的收益，但传统重要性采样方法的方差随时间步呈指数级爆炸，实用性很差。近年来兴起了一类基于函数近似的极小极大（minimax）方法，分为"学习重要性权重"和"学习价值函数"两种流派，虽然缓解了方差问题，却存在两大痛点：函数近似带来的偏差难以量化，且两种流派各自只利用了数据的部分信息、彼此割裂。本文通过对两种流派的代表方法稍加改造，将它们统一推导为一个**价值区间**（value interval），该区间具有一种特殊的双重鲁棒性——只要价值函数类或重要性权重类中有一个被正确指定，区间就是有效的，而区间的宽度则恰好量化了另一个类的误设程度；此外，每个界都来自一个同时利用奖励、初始状态等所有数据成分的单一优化问题，比以往方法更紧。这一框架不仅统一了多个近期方法并揭示其内在联系，还自然地对应了离线策略优化中经典的悲观原则（用于数据覆盖不足时的利用）和乐观原则（用于探索），为可信赖的强化学习评估与优化提供了更坚实的理论基础。
原文 arXiv:2002.02081;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.02081v6