Learning to Communicate with Deep Multi-Agent Reinforcement Learning
Jakob N. Foerster1,†、Yannis M. Assael1,†、Nando de Freitas1,2,3、Shimon Whiteson1 \AND1University of Oxford, United Kingdom 2Canadian Institute for Advanced Research, CIFAR NCAP Program 3Google DeepMind
Abstract
We consider the problem of multiple agents sensing and acting in environments with the goal of maximising their shared utility. In these environments, agents must learn communication protocols in order to share information that is needed to solve the tasks. By embracing deep neural networks, we are able to demonstrate end-to-end learning of protocols in complex environments inspired by communication riddles and multi-agent computer vision problems with partial observability. We propose two approaches for learning in these domains: Reinforced Inter-Agent Learning (RIAL) and Differentiable Inter-Agent Learning (DIAL). The former uses deep Q-learning, while the latter exploits the fact that, during learning, agents can backpropagate error derivatives through (noisy) communication channels. Hence, this approach uses centralised learning but decentralised execution. Our experiments introduce new environments for studying the learning of communication protocols and present a set of engineering innovations that are essential for success in these domains.
中文速览
多智能体系统中，如果每个机器人只能看到局部信息，它们就必须互相"说话"才能协作完成任务——但如何让智能体自动学会一套有效的通信语言，而不是靠人工设计，一直是个难题。这篇论文提出了两种端到端的学习方法：RIAL（强化智能体间学习）基于深度Q学习，让智能体像强化学习一样摸索出沟通方式；DIAL（可微智能体间学习）则更进一步，在集中训练阶段允许误差梯度直接"穿越"通信信道在智能体之间传播，相当于把通信本身也变成神经网络的一部分，训练完成后再切换到离散信号做分布式执行。实验在受通信谜题和多智能体计算机视觉启发的新基准任务上验证了这两种方法，结果显示DIAL因为能跨智能体反向传播而表现更优，智能体往往能自发涌现出简洁优雅的通信协议。这项工作第一次证明深度学习可以在复杂序列任务中从零习得智能体间的通信协议，为研究语言涌现和多智能体协作提供了可扩展的技术路径。
原文 arXiv:1605.06676;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1605.06676v2