Counterfactual Multi-Agent Policy Gradients
Jakob N. Foerster1,†、Gregory Farquhar1,† \ANDTriantafyllos Afouras1、Nantas Nardelli1、Shimon Whiteson1 \AND1University of Oxford, United Kingdom †Equal contribution
Abstract
Many real-world problems, such as network packet routing and the coordination of autonomous vehicles, are naturally modelled as cooperative multi-agent systems. There is a great need for new reinforcement learning methods that can efficiently learn decentralised policies for such systems. To this end, we propose a new multi-agent actor-critic method called counterfactual multi-agent (COMA) policy gradients. COMA uses a centralised critic to estimate the $Q$ -function and decentralised actors to optimise the agents’ policies. In addition, to address the challenges of multi-agent credit assignment, it uses a counterfactual baseline that marginalises out a single agent’s action, while keeping the other agents’ actions fixed. COMA also uses a critic representation that allows the counterfactual baseline to be computed efficiently in a single forward pass. We evaluate COMA in the testbed of StarCraft unit micromanagement, using a decentralised variant with significant partial observability. COMA significantly improves average performance over other multi-agent actor-critic methods in this setting, and the best performing agents are competitive with state-of-the-art centralised controlle
中文速览
多智能体协作系统中，所有智能体共享同一个全局奖励，导致每个智能体很难判断自己的行为究竟对团队贡献了多少——这就是"功劳分配"难题。为了解决这个问题，研究者提出了反事实多智能体策略梯度方法（COMA，Counterfactual Multi-Agent Policy Gradients）：训练时用一个集中式的评价网络（critic）来估计联合动作的价值，执行时每个智能体只依靠自己的局部观测独立决策；核心创新在于引入"反事实基线"——把某个智能体的动作换成其他所有可能动作的期望值，同时固定其他智能体的动作不变，从而计算出该智能体独有的优势函数，精准衡量它的实际贡献；此外，通过专门设计的网络结构，所有动作的Q值可以在一次前向传播中同时算出，大幅提升效率。在《星际争霸》单位微操这一高难度强化学习基准测试中，COMA明显优于其他多智能体演员-评价方法，最优智能体的表现甚至可与能获取全局状态信息的集中式控制器媲美，证明了在部分可观测、去中心化执行的现实场景下，这一方法具有重要的实用价值。
原文 arXiv:1705.08926;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1705.08926v3