Increasing the Action Gap: New Operators for Reinforcement Learning
Marc G. Bellemare and Georg Ostrovski and Arthur Guez Philip S. Thomas and Rémi Munos Google DeepMind Now at Carnegie Mellon University.
Abstract
This paper introduces new optimality-preserving operators on Q-functions. We first describe an operator for tabular representations, the consistent Bellman operator, which incorporates a notion of local policy consistency. We show that this local consistency leads to an increase in the action gap at each state; increasing this gap, we argue, mitigates the undesirable effects of approximation and estimation errors on the induced greedy policies. This operator can also be applied to discretized continuous space and time problems, and we provide empirical results evidencing superior performance in this context. Extending the idea of a locally consistent operator, we then derive sufficient conditions for an operator to preserve optimality, leading to a family of operators which includes our consistent Bellman operator. As corollaries we provide a proof of optimality for Baird’s advantage learning algorithm and derive other gap-increasing operators with interesting properties. We conclude with an empirical study on 60 Atari 2600 games illustrating the strong potential of these new operators.
中文速览
强化学习（reinforcement learning）中，标准Bellman算子在面对次优动作时描述的是一种非稳态策略的价值，这会导致最优动作与次优动作之间的"行动间隙（action gap）"偏小，使得在函数逼近或估计误差存在时，贪心策略容易选错动作。为此，论文提出了"一致Bellman算子（consistent Bellman operator）"，通过在局部强制动作选择的一致性来压低次优动作的Q值，从而拉大行动间隙，同时严格保证最优策略不变。在此基础上，作者进一步推导出一大类保最优性算子的充分条件，证明这类算子无需是压缩映射、甚至无需保证次优动作的Q值收敛，并以此为Baird的优势学习算法提供了严格的最优性证明。在实验上，将DQN的学习规则替换为新算子后，仅改动"一行代码"便在60款Atari 2600游戏上取得了显著超越原始DQN的成绩，说明重新审视值迭代的核心组件具有重要的实践价值。
原文 arXiv:1512.04860;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1512.04860v1