Playing hard exploration games by watching YouTube
Yusuf Aytar , Tobias Pfaff*, David Budden, Tom Le Paine, Ziyu Wang, Nando de Freitas DeepMind, London, UK denotes equal contribution
Abstract
Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent’s exact environment setup and the demonstrator’s action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games Montezuma’s Revenge, Pitfall! and Private Eye for the first time, even if the agent is not presented with any environment rewards.
中文速览
用一段YouTube游戏录像教AI打通"以难出名"的雅达利游戏——这就是这项研究的核心目标。研究者面对的挑战是：网上的游戏视频画质各异、音画不同步，既没有玩家的操作记录，也没有奖励信号，传统模仿学习方法根本无从下手。为此，他们设计了两步走的方案：第一步，用"时序距离分类（TDC）"和"跨模态时序距离分类（CMC）"两个自监督任务，让网络同时学习来自多个来源的视频和音频，从而把嘈杂、不对齐的多域数据映射到同一个表示空间；第二步，把一段YouTube玩家录像嵌入这个空间，构造出一个模仿奖励信号，驱动强化学习智能体去复现人类的游戏路径。最终，该方法让AI在业界公认最难探索的三款游戏——《蒙特祖玛的复仇》《陷阱！》和《私家侦探》——上首次全部突破人类水平，甚至在完全没有环境奖励的情况下也能做到，证明了仅凭一段普通网络视频就能有效指导强化学习智能体的巨大潜力。
原文 arXiv:1805.11592;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1805.11592v2