Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments
Ryan Lowe∗ McGill University OpenAI、Yi Wu UC Berkeley、Aviv Tamar UC Berkeley \ANDJean Harb McGill University OpenAI、Pieter Abbeel UC Berkeley OpenAI、Igor Mordatch OpenAI Equal contribution.
Abstract
We explore deep reinforcement learning methods for multi-agent domains. We begin by analyzing the difficulty of traditional algorithms in the multi-agent case: Q-learning is challenged by an inherent non-stationarity of the environment, while policy gradient suffers from a variance that increases as the number of agents grows. We then present an adaptation of actor-critic methods that considers action policies of other agents and is able to successfully learn policies that require complex multi-agent coordination. Additionally, we introduce a training regimen utilizing an ensemble of policies for each agent that leads to more robust multi-agent policies. We show the strength of our approach compared to existing methods in cooperative as well as competitive scenarios, where agent populations are able to discover various physical and informational coordination strategies.
中文速览
多智能体环境中的强化学习面临两大顽疾：Q-learning因环境非平稳性而难以收敛，策略梯度（policy gradient）的方差又随智能体数量增加而急剧膨胀。针对这两个问题，研究者提出了一种"集中训练、分散执行"的多智能体演员-评论家算法（Multi-Agent Actor-Critic，MADDPG）——训练时让每个智能体的评论家（critic）获取所有智能体的动作和策略信息，执行时每个智能体仅依靠自身本地观测独立决策。此外，该方法还引入了策略集成（policy ensemble）训练机制，让每个智能体与多种不同的对手或队友策略交互，从而提升鲁棒性。在合作、竞争及混合场景的实验中，该方法均显著优于独立学习的基线，智能体能自发涌现出物理协调与信息沟通等复杂策略。这项工作的重要性在于，它提供了一个无需可微环境模型、无需特定通信结构的通用多智能体学习框架，为构建能与人类及其他智能体高效协作的AI系统奠定了基础。
原文 arXiv:1706.02275;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.02275v4