Deep Reinforcement Learning at the Edge of the Statistical Precipice
Rishabh Agarwal Google Research, Brain Team MILA, Université de Montréal、Max Schwarzer MILA, Université de Montréal、Pablo Samuel Castro Google Research, Brain Team Aaron Courville MILA, Université de Montréal、Marc G. Bellemare Google Research, Brain Team Outstanding Paper Award. Correspondence to Rishabh
Abstract
Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few-run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field’s confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance
中文速览
深度强化学习（deep RL）领域长期以来用"点估计"（如均值、中位数）来比较算法性能，却忽视了因训练次数极少（通常只有3–10次）而带来的巨大统计不确定性。本文以Atari 100k基准为案例，揭示了这种做法会导致结论严重失真——在极少次数运行下，点估计的随机波动足以颠覆算法之间的优劣排序，甚至让更差的算法看起来更好。为此，作者提出一套更严谨的评估方法：用分层自助法（stratified bootstrap）报告置信区间而非单一数值，用"四分位均值"（interquartile mean, IQM）替代易受极端值干扰的均值和高方差的中位数，并引入性能分布曲线（performance profiles）来展示算法在全部任务上的表现分布。将这套方法应用于ALE、Procgen和DeepMind Control Suite等主流基准后，作者再次发现既有比较结论存在明显偏差。这项工作及其配套开源库rliable呼吁整个深度RL社区改变评估习惯，以避免不可靠的结果误导研究方向、阻碍领域真正进步。
原文 arXiv:2108.13264;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2108.13264v4