Intrinsic Motivation and Automatic Curricula via Asymmetric Self-Play
Sainbayar Sukhbaatar Dept. of Computer Science New York University、Zeming Lin Facebook AI Research New York、Ilya Kostrikov Dept. of Computer Science New York University、Gabriel Synnaeve, Arthur Szlam、Rob Fergus Facebook AI Research New York
Abstract
We describe a simple scheme that allows an agent to learn about its environment in an unsupervised manner. Our scheme pits two versions of the same agent, Alice and Bob, against one another. Alice proposes a task for Bob to complete; and then Bob attempts to complete the task. In this work we will focus on two kinds of environments: (nearly) reversible environments and environments that can be reset. Alice will “propose” the task by doing a sequence of actions and then Bob must undo or repeat them, respectively. Via an appropriate reward structure, Alice and Bob automatically generate a curriculum of exploration, enabling unsupervised training of the agent. When Bob is deployed on an RL task within the environment, this unsupervised training reduces the number of supervised episodes needed to learn, and in some cases converges to a higher reward.
中文速览
让智能体在没有任何外部奖励的情况下自主探索并掌握环境规律，是强化学习长期面临的难题。研究者提出了一种"自我博弈"（self-play）框架：将同一个智能体拆分为"爱丽丝"和"鲍勃"两个角色，爱丽丝先在环境中执行一段动作序列来"出题"，鲍勃再尝试还原或重复这段轨迹来"答题"；通过精心设计的内部奖励机制，爱丽丝会自动找到刚好超出鲍勃当前能力边界的任务，从而在无需人工干预的情况下为鲍勃生成一套由易到难的自适应课程。在迷宫、连续控制和星际争霸等多类环境的实验中，经过自我博弈预训练的鲍勃在面对真正的有监督强化学习任务时，所需的有奖励样本数量显著减少，部分任务还能收敛到更高的最终回报。这项工作的意义在于：它提供了一种纯粹无监督、无需人工设计课程的方式来让智能体"练基本功"，为解决稀疏奖励和样本效率低下问题提供了一条切实可行的新路径。
原文 arXiv:1703.05407;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.05407v5