Stable Opponent Shaping in Differentiable Games
Alistair Letcher1 Jakob Foerster1 David Balduzzi2 Tim Rocktäschel3 Shimon Whiteson1 1University of Oxford 2DeepMind 3University College London
Abstract
A growing number of learning methods are actually differentiable games whose players optimise multiple, interdependent objectives in parallel – from GANs and intrinsic curiosity to multi-agent RL. Opponent shaping is a powerful approach to improve learning dynamics in these games, accounting for player influence on others’ updates. Learning with Opponent-Learning Awareness (LOLA) is a recent algorithm that exploits this response and leads to cooperation in settings like the Iterated Prisoner’s Dilemma. Although experimentally successful, we show that LOLA agents can exhibit ‘arrogant’ behaviour directly at odds with convergence. In fact, remarkably few algorithms have theoretical guarantees applying across all ( $n$ -player, non-convex) games. In this paper we present Stable Opponent Shaping (SOS), a new method that interpolates between LOLA and a stable variant named LookAhead. We prove that LookAhead converges locally to equilibria and avoids strict saddles in all differentiable games. SOS inherits these essential guarantees, while also shaping the learning of opponents and consistently either matching or outperforming LOLA experimentally.
中文速览
多目标并行优化的机器学习系统（如GAN、多智能体强化学习）本质上是"可微博弈"，现有算法要么缺乏理论收敛保证，要么虽有保证却丧失了对对手学习过程的"塑造"能力。论文首先指出当前最流行的对手感知算法LOLA（Learning with Opponent-Learning Awareness）会表现出"傲慢"行为——在某些博弈中偏离均衡点、损害所有参与者的利益；随后提出LookAhead算法并从理论上严格证明它在所有可微博弈（任意玩家数、非凸）中都能局部收敛到稳定均衡并绕开不稳定鞍点，填补了多智能体学习领域的重要理论空白；在此基础上，作者进一步提出结合LookAhead稳定性与LOLA对手塑造能力的新算法SOS（Stable Opponent Shaping），通过一个有理论依据的插值参数在两者之间动态权衡。实验表明SOS在囚徒困境中与LOLA一样能涌现出"以牙还牙"的合作策略，在Tandem博弈和GAN训练等场景中还稳定超越LOLA，这对于构建既有理论保障又实用高效的多智能体学习系统具有重要意义。
原文 arXiv:1811.08469;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.08469v3