Doubly robust off-policy evaluation with shrinkage
Yi Su Maria Dimakopoulou Akshay Krishnamurthy Miroslav Dudík
Abstract
We propose a new framework for designing estimators for off-policy evaluation in contextual bandits. Our approach is based on the asymptotically optimal doubly robust estimator, but we shrink the importance weights to minimize a bound on the mean squared error, which results in a better bias-variance tradeoff in finite samples. We use this optimization-based framework to obtain three estimators: (a) a weight-clipping estimator, (b) a new weight-shrinkage estimator, and (c) the first shrinkage-based estimator for combinatorial action sets. Extensive experiments in both standard and combinatorial bandit benchmark problems show that our estimators are highly adaptive and typically outperform state-of-the-art methods.
中文速览
离线策略评估（off-policy evaluation）是个棘手难题：我们手头只有旧策略收集的数据，却想准确估算新策略的表现，而现有的双重鲁棒（doubly robust, DR）估计量虽然渐近最优，但在样本有限时，一旦重要性权重（importance weight）过大，方差就会急剧膨胀。这篇论文提出了一套系统性的"权重收缩"框架——通过直接最小化均方误差的上界，把重要性权重压缩到更合适的量级，从而在偏差和方差之间取得更好的平衡。基于这套框架，作者推导出三类估计量：一个对经典截断法给出了理论解释，一个利用奖励预测器质量自适应调整权重（乐观收缩），还有一个首次将收缩思路推广到组合动作集合（如同时推荐多篇文章的场景）。在涵盖9个真实数据集、108种实验条件的大规模对比实验中，新估计量普遍优于现有最优方法，为实际推荐、广告等场景的离线策略评估提供了更稳健、更实用的工具。
原文 arXiv:1907.09623;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.09623v2