Open-Ended Learning Leads to Generally Capable Agents
Open-Ended Learning Team Adam Stooke Anuj Mahajan Catarina Barros Charlie Deck Jakob Bauer Jakub Sygnowski Maja Trebacz Max Jaderberg Michael Mathieu Nat McAleese Nathalie Bradley-Schmieg Nathaniel Wong Nicolas Porcel Roberta Raileanu Steph Hughes-Fitt Valentin Dalibard Wojciech Marian Czarnecki
Abstract
Artificial agents have achieved great success in individual challenging simulated environments, mastering the particular tasks they were trained for, with their behaviour even generalising to maps and opponents that were never encountered in training. In this work we create agents that can perform well beyond a single, individual task, that exhibit much wider generalisation of behaviour to a massive, rich space of challenges. We define a universe of tasks within an environment domain and demonstrate the ability to train agents that are generally capable across this vast space and beyond. The environment is natively multi-agent, spanning the continuum of competitive, cooperative, and independent games, which are situated within procedurally generated physical 3D worlds. The resulting space is exceptionally diverse in terms of the challenges posed to agents, and as such, even measuring the learning progress of an agent is an open research problem. We propose an iterative notion of improvement between successive generations of agents, rather than seeking to maximise a singular objective, allowing us to quantify progress despite tasks being incomparable in terms of achievable rewards.
中文速览
训练一个能在海量不同任务上都表现良好的通用智能体，而不是只会某一种特定游戏，是强化学习领域长期未解决的难题。研究者构建了一个名为 XLand 的三维多智能体环境，通过程序化生成数十亿种不同的物理世界与多人游戏规则组合，覆盖竞争、合作、混合等各类场景，形成一个极为庞大且多样的任务空间；在此基础上，他们设计了一套开放式训练流程（open-ended learning），动态调整训练任务分布和对手策略，配合多代智能体的蒸馏迭代，让智能体始终处于"刚好有挑战"的学习状态，而非被固定任务分布所局限。最终训练出的智能体能在所有人类可解的评测关卡中获得奖励，并在从未见过的捉迷藏、夺旗、标记等任务上展现出零样本泛化能力，还涌现出试错探索、简单工具使用、与他人合作等启发式行为，且通过极少量微调即可在新任务上大幅超越从头训练的基线。这项工作表明，开放式动态课程结合大规模多任务训练，是实现真正意义上通用智能体的一条可行路径。
原文 arXiv:2107.12808;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2107.12808v2