Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents
\nameMarlos C. Machado \addrUniversity of Alberta, Edmonton, Canada \AND\nameMarc G. Bellemare \addrGoogle Brain, Montréal, Canada \AND\nameErik Talvitie \addrFranklin、Marshall College, Lancaster, USA \AND\nameJoel \addrDeepMind, London, United Kingdom \AND\nameMatthew \addrMicrosoft Research, Redmond, USA \AND\nameMichael Bowling \addrUniversity of Alberta, Edmonton, Canada \addrDeepMind, Edmonton, Canada Work performed at DeepMind.
Abstract
The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.
中文速览
作为评估AI通用能力的主流平台，Atari游戏环境（Arcade Learning Environment，ALE）近年来被大量研究引用，但各团队采用的评测方法五花八门、互不统一，导致结果之间难以直接比较。为此，本文系统梳理了文献中存在的各种评测差异，归纳出一套方法论最佳实践，并在统一标准下给出了新的基准测试结果，方便后续研究直接对标。在技术层面，文章指出旧版ALE完全确定性的动态机制会让智能体通过死记硬背动作序列"走捷径"，因此引入了"粘滞动作"（sticky actions）这一随机性机制，并新增多游戏模式支持，让评测环境更贴近真实挑战。最终，文章还回顾了ALE最初提出时列出的各项开放问题，总结了当前技术进展，指出哪些挑战已被攻克、哪些仍悬而未决。这项工作的意义在于为整个领域校准方向：规范评测标准不仅能让已有成果的比较更加公平可信，也能为未来通用AI能力测试平台的建设提供可借鉴的方法论基础。
原文 arXiv:1709.06009;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.06009v2