Mastering Atari with Discrete World Models
Danijar Hafner Google Research、Timothy Lillicrap DeepMind、Mohammad Norouzi Google Research、Jimmy Ba University of Toronto Correspondence to: Danijar Hafner
Abstract
Intelligent agents need to generalize from past experience to achieve goals in complex environments. World models facilitate such generalization and allow learning behaviors from imagined outcomes to increase sample-efficiency. While learning world models from image inputs has recently become feasible for some tasks, modeling Atari games accurately enough to derive successful behaviors has remained an open challenge for many years. We introduce DreamerV2, a reinforcement learning agent that learns behaviors purely from predictions in the compact latent space of a powerful world model. The world model uses discrete representations and is trained separately from the policy. DreamerV2 constitutes the first agent that achieves human-level performance on the Atari benchmark of 55 tasks by learning behaviors inside a separately trained world model. With the same computational budget and wall-clock time, Dreamer V2 reaches 200M frames and surpasses the final performance of the top single-GPU agents IQN and Rainbow. DreamerV2 is also applicable to tasks with continuous actions, where it learns an accurate world model of a complex humanoid robot and solves stand-up and walking from only pix
中文速览
为了让强化学习智能体从图像像素中高效学会玩Atari游戏，研究者提出了DreamerV2——一个完全在自己构建的"世界模型"内部想象、规划并学习行为策略的智能体。核心方法是用离散的类别变量（categorical latents）替代此前的高斯连续隐变量来表示环境状态，同时引入"KL平衡"技巧让世界模型的先验分布学得更准，使得智能体在脑海中模拟出的轨迹足够可靠，无需真实环境交互就能训练出有效策略。实验结果表明，DreamerV2在Atari 55款游戏基准上首次以纯世界模型内学习的方式达到人类水平，在相同算力（单张GPU）和时钟时间下，最终得分超过了Rainbow和IQN等经过多年迭代的无模型顶级算法。这项工作证明世界模型已经强大到足以与最优秀的无模型方法正面竞争，为样本高效、可泛化的强化学习研究开辟了新方向。
原文 arXiv:2010.02193;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.02193v4