Observational Learning by Reinforcement Learning
Diana Borsa DeepMind、Bilal Piot DeepMind \ANDRémi Munos DeepMind、Olivier Pietquin DeepMind
Abstract
Observational learning is a type of learning that occurs as a function of observing, retaining and possibly replicating or imitating the behaviour of another agent. It is a core mechanism appearing in various instances of social learning and has been found to be employed in several intelligent species, including humans. In this paper, we investigate to what extent the explicit modelling of other agents is necessary to achieve observational learning through machine learning. Especially, we argue that observational learning can emerge from pure Reinforcement Learning (RL), potentially coupled with memory. Through simple scenarios, we demonstrate that an RL agent can leverage the information provided by the observations of an other agent performing a task in a shared environment. The other agent is only observed through the effect of its actions on the environment and never explicitly modeled. Two key aspects are borrowed from observational learning: i) the observer behaviour needs to change as a result of viewing a ’teacher’ (another agent) and ii) the observer needs to be motivated somehow to engage in making use of the other agent’s behaviour. The later is naturally modeled by RL,
中文速览
观察其他智能体行为就能学习——不需要明确建模对方、不需要克隆对方动作、不需要推断对方奖励——这件事在强化学习（Reinforcement Learning, RL）框架下到底能不能自然发生？研究者设计了一系列从简单到复杂的共享环境场景，让一个基于A3C算法、配备记忆模块（LSTM）的学习智能体和一个"教师"智能体共存，学习者只能通过观察环境状态的变化间接感知教师的行为，既不知道对方是另一个智能体，也不会得到任何额外的模仿奖励。实验结果表明，仅靠环境奖励信号与记忆能力的结合，学习者便能自发涌现出模仿（imitation）和非模仿式社会学习（non-imitative social learning）等复杂行为，在信息不完整的情况下借助教师的示范显著提升策略，并在教师离开后依然保持独立完成任务的能力。这一发现说明，无需繁琐的专家轨迹对齐或奖励函数推断，纯粹的深度强化学习加记忆就足以让机器实现类人的观察学习，为构建更自然、更通用的多智能体学习系统提供了新的理论依据和实验范式。
原文 arXiv:1706.06617;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.06617v1