Agent57: Outperforming the Atari Human Benchmark
Adrià Puigdomènech Badia Bilal Piot Steven Kapturowski Pablo Sprechmann Alex Vitvitskyi Daniel Guo Charles Blundell
Abstract
Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.
中文速览
强化学习领域长期存在一个未竟难题：尽管各种算法在57款Atari游戏的平均表现上屡创新高，却始终有几款极具挑战性的游戏（如需要长时序信用分配的Skiing，以及探索极度稀疏的Montezuma's Revenge）无法超越人类基准。为此，研究者在已有的NGU智能体基础上提出了Agent57，核心改进有两点：一是将状态-动作价值函数拆分为内在奖励和外在奖励两个独立分支，大幅提升了训练稳定性；二是引入一个基于多臂老虎机的元控制器，让智能体在训练过程中自适应地分配不同探索程度策略的资源，而非对所有策略一视同仁地均匀采样。最终，Agent57成为首个在全部57款Atari游戏上同时超越人类基准的深度强化学习智能体，填补了该领域十年来的空白，标志着通用游戏智能体在这一经典测试平台上达到了真正意义上的全面过关。
原文 arXiv:2003.13350;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.13350v1