Learning with AMIGo: Adversarially Motivated Intrinsic Goals
Andres Campero Brain and Cognitive Sciences, MIT Cambridge, USA、Roberta Raileanu New York University New York, USA、Heinrich Küttler Facebook AI Research London, UK、Joshua B. Tenenbaum Brain and Cognitive Sciences, MIT Cambridge, USA、Tim Rocktäschel University College London、Facebook AI Research London, UK、Edward Grefenstette University College London、Facebook AI Research London, UK Work done during an internship at Facebook AI Research.
Abstract
A key challenge for reinforcement learning (RL) consists of learning in environments with sparse extrinsic rewards. In contrast to current RL methods, humans are able to learn new skills with little or no reward by using various forms of intrinsic motivation. We propose AMIGo, a novel agent incorporating—as form of meta-learning—a goal-generating teacher that proposes Adversarially Motivated Intrinsic Goals to train a goal-conditioned “student” policy in the absence of (or alongside) environment reward. Specifically, through a simple but effective “constructively adversarial” objective, the teacher learns to propose increasingly challenging—yet achievable—goals that allow the student to learn general skills for acting in a new environment, independent of the task to be solved. We show that our method generates a natural curriculum of self-proposed goals which ultimately allows the agent to solve challenging procedurally-generated tasks where other forms of intrinsic motivation and state-of-the-art RL methods fail.
中文速览
在稀疏奖励环境中训练强化学习（RL）智能体极为困难，因为随机探索几乎不可能偶然碰到有意义的奖励信号。为此，研究者提出了 AMIGo（Adversarially Motivated Intrinsic Goals，对抗驱动的内在目标）方法：将智能体拆分为一个"教师"和一个"学生"，教师不断为学生生成难度适中的目标——既不太简单也不太难——从而自动形成一套由易到难的课程，驱动学生持续探索和学习环境技能。教师的训练目标设计得十分巧妙：只有当学生需要花费适度步数才能完成目标时教师才获得奖励，这迫使双方相互"对抗"又相互促进。在 MiniGrid 这套公认具有挑战性的程序化生成环境基准上，AMIGo 在多个最难关卡上成为唯一能获得外部奖励的方法，超越了包括好奇心驱动探索在内的多种前沿内在激励方案。这项工作表明，用类似儿童"自我设定挑战"的机制来替代盲目探索，是解决稀疏奖励问题的一条切实可行的道路。
原文 arXiv:2006.12122;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.12122v2