Actor-Mimic Deep Multitask and Transfer Reinforcement Learning
Emilio Parisotto, Jimmy Ba, Ruslan Salakhutdinov Department of Computer Science University of Toronto Toronto, Ontario, Canada
Abstract
The ability to act in multiple environments and transfer previous knowledge to new situations can be considered a critical aspect of any intelligent agent. Towards this goal, we define a novel method of multitask and transfer learning that enables an autonomous agent to learn how to behave in multiple tasks simultaneously, and then generalize its knowledge to new domains. This method, termed “Actor-Mimic”, exploits the use of deep reinforcement learning and model compression techniques to train a single policy network that learns how to act in a set of distinct tasks by using the guidance of several expert teachers. We then show that the representations learnt by the deep policy network are capable of generalizing to new tasks with no prior expert guidance, speeding up learning in novel environments. Although our method can in general be applied to a wide range of problems, we use Atari games as a testing environment to demonstrate these methods.
中文速览
用多任务学习和迁移学习让一个神经网络同时玩好多款 Atari 游戏，并把学到的知识迁移到新游戏上——这是这项工作要解决的核心问题。研究者提出了"Actor-Mimic"方法：先为每款源游戏单独训练一个专家网络（DQN），再让一个统一的多任务网络同时向所有专家"拜师学艺"，通过模仿专家的行动策略（策略回归）和模仿专家的内部特征表示（特征回归）来压缩并融合多个专家的知识。迁移时，只需把这个多任务网络的权重作为新游戏 DQN 的初始参数，结果表明它能在陌生游戏上显著加快学习速度，而不需要任何针对新游戏的专家指导。这项工作的意义在于：它打破了深度强化学习"一个网络只能玩一个游戏"的局限，向真正具备跨任务泛化能力的智能体迈出了重要一步，同时大幅降低了在新任务上的训练成本。
原文 arXiv:1511.06342;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1511.06342v4