Minimax Weight and Q-Function Learning for Off-Policy Evaluation
Masatoshi Uehara Jiawei Huang Nan Jiang
Abstract
We provide theoretical investigations into off-policy evaluation in reinforcement learning using function approximators for (marginalized) importance weights and value functions. Our contributions include: (1) A new estimator, MWL, that directly estimates importance ratios over the state-action distributions, removing the reliance on knowledge of the behavior policy as in prior work (Liu et al., 2018). (2) Another new estimator, MQL, obtained by swapping the roles of importance weights and value-functions in MWL. MQL has an intuitive interpretation of minimizing average Bellman errors and can be combined with MWL in a doubly robust manner. (3) Several additional results that offer further insights, including the sample complexities of MWL and MQL, their asymptotic optimality in the tabular setting, how the learned importance weights depend the choice of the discriminator class, and how our methods provide a unified view of some old and new algorithms in RL.
中文速览
离线策略评估（off-policy evaluation）要解决的是：如何用一批旧策略收集的历史数据，可靠地估算新策略的期望回报——这在现实中极为重要，因为直接跑新策略往往成本高昂甚至危险。已有的边际重要性采样（marginalized importance sampling）方法虽然克服了传统重要性采样"随时间步指数级爆炸"的方差问题，但仍依赖对行为策略的精确知识，且理论保证不够完善。本文提出两个新估计量：MWL 直接在状态-动作联合分布上学习重要性权重，无需知道行为策略；MQL 则通过交换重要性权重与 Q 函数的角色，将问题转化为最小化平均贝尔曼误差，两者还可以"双重鲁棒"方式结合，在任一个估计量准确时整体估计仍然有效。理论分析表明，这两个方法在表格（tabular）设置下都能达到半参数（semiparametric）统计下界，即渐近最优，并将 LSTD、逐步重要性采样、基于模型的离线评估等经典算法统一到同一框架下，为离线强化学习的理论与算法设计提供了坚实基础。
原文 arXiv:1910.12809;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.12809v4