Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control
Riashat Islam School of Computer Science McGill University Montreal, QC, Canada \AndPeter Henderson††School of Computer Science McGill University Montreal, QC, Canada \AndMaziar Gomrokchi School of Computer Science McGill University Montreal, QC, Canada \AndDoina Precup School of Computer Science McGill University Montreal, QC, Canada Work done while interning at Maluuba, A Microsoft Company.These two authors contributed equally.
Abstract
Policy gradient methods in reinforcement learning have become increasingly prevalent for state-of-the-art performance in continuous control tasks. Novel methods typically benchmark against a few key algorithms such as deep deterministic policy gradients and trust region policy optimization. As such, it is important to present and use consistent baselines experiments. However, this can be difficult due to general variance in the algorithms, hyper-parameter tuning, and environment stochasticity. We investigate and discuss: the significance of hyper-parameters in policy gradients for continuous control, general variance in the algorithms, and reproducibility of reported results. We provide guidelines on reporting novel results as comparisons against baseline methods such that future researchers can make informed decisions when investigating novel methods.
中文速览
策略梯度（policy gradient）方法在连续控制任务中表现出色，但研究者在复现和比较 DDPG、TRPO 等基线算法时往往面临结果差异巨大的困境。作者系统测试了网络结构、批量大小、步长、奖励缩放等多种超参数对算法性能的影响，并在相同超参数下用不同随机种子重复实验，发现仅凭随机种子的差异就能让同一算法产生在统计上截然不同的学习曲线。研究还揭示了文献中常用的"最大回报"等指标存在严重偏差，而不同论文使用不一致的超参数组合作为基线，会让算法间的比较严重失真。这项工作提醒深度强化学习社区：在报告新方法效果时，必须统一超参数设置、提供充足的重复实验次数并采用无偏评估指标，否则所谓的"性能提升"很可能只是实验设置不公平的产物。
原文 arXiv:1708.04133;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1708.04133v1