Learning with Opponent-Learning Awareness
Jakob Foerster†,‡ , Richard Y. Chen† , Maruan Al-Shedivat‡ , Shimon Whiteson , Pieter Abbeel‡ and Igor Mordatch
Abstract
Multi-agent settings are quickly gathering importance in machine learning. This includes a plethora of recent work on deep multi-agent reinforcement learning, but also can be extended to hierarchical reinforcement learning, generative adversarial networks and decentralised optimization. In all these settings the presence of multiple learning agents renders the training problem non-stationary and often leads to unstable training or undesired final results. We present Learning with Opponent-Learning Awareness (LOLA), a method in which each agent shapes the anticipated learning of the other agents in the environment. The LOLA learning rule includes an additional term that accounts for the impact of one agent’s policy on the anticipated parameter update of the other agents. Preliminary results show that the encounter of two LOLA agents leads to the emergence of tit-for-tat and therefore cooperation in the iterated prisoners’ dilemma (IPD), while independent learning does not. In this domain, LOLA also receives higher payouts compared to a naive learner, and is robust against exploitation by higher order gradient-based methods. Applied to infinitely repeated matching pennies, LOLA agent
中文速览
多智能体强化学习中，多个智能体同时学习会让训练环境变得不稳定，现有方法通常把对手当成静态环境的一部分来处理，结果在"囚徒困境"这样的博弈中，自私的智能体几乎必然走向相互背叛。本文提出了 LOLA（带对手学习感知的学习，Learning with Opponent-Learning Awareness）方法：每个智能体在更新自身策略时，额外考虑"我的行动会如何影响对手下一步的学习方向"，也就是对对手的梯度更新步骤再做一次求导，主动塑造对手的学习轨迹。实验表明，两个 LOLA 智能体在迭代囚徒困境中能自发涌现出"以牙还牙"的互惠合作策略，而独立学习的基准方法做不到这一点；在无限重复的匹配猜硬币博弈中，LOLA 智能体也能收敛到纳什均衡；在嵌入社会困境的网格世界任务中，借助循环神经网络策略和对手建模，LOLA 同样实现了合作。这项工作的重要性在于，它首次让智能体通过纯粹的自利学习规则自然涌现出互惠合作，而非依赖硬编码的启发式策略，为未来人工智能体与人类共存的复杂社会场景提供了更通用的合作机制。
原文 arXiv:1709.04326;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.04326v4