Multi-task Deep Reinforcement Learning with PopArt
Matteo Hessel、Hubert Soyer、Lasse Espeholt \ANDWojciech Czarnecki、Simon Schmitt、Hado van Hasselt
Abstract
The reinforcement learning community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequential-decision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent’s updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to p
中文速览
训练一个强化学习（RL）智能体同时精通几十款差异巨大的游戏，核心难题在于不同任务的奖励幅度和稀疏程度天差地别，导致高奖励任务"抢占"学习资源，低奖励任务被忽视。研究者将一种名为PopArt的自适应归一化方法引入多任务actor-critic框架，让每个任务的价值估计和策略梯度更新都被自动缩放到相近量级，从而让所有任务对智能体参数的影响力趋于均衡。在57款Atari游戏的多任务基准上，用单套网络权重训练出的单一策略达到了人类表现中位数的110%，成为首个在该多任务设置下超越人类水平的智能体；在DeepMind Lab的30个3D任务上也同样刷新了当时的最优成绩。这项工作表明，解决多任务RL中的"任务显著性失衡"问题，仅靠优雅的归一化设计而非复杂的架构改动就能带来质的飞跃，对构建真正通用的RL智能体具有重要意义。
原文 arXiv:1809.04474;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.04474v1