Deep Reinforcement Learning from Self-Play in Imperfect-Information Games
Johannes Heinrich University College London, UK、David Silver University College London, UK
Abstract
Many real-world applications can be described as large-scale games of imperfect information. To deal with these challenging domains, prior work has focused on computing Nash equilibria in a handcrafted abstraction of the domain. In this paper we introduce the first scalable end-to-end approach to learning approximate Nash equilibria without prior domain knowledge. Our method combines fictitious self-play with deep reinforcement learning. When applied to Leduc poker, Neural Fictitious Self-Play (NFSP) approached a Nash equilibrium, whereas common reinforcement learning methods diverged. In Limit Texas Hold’em, a poker game of real-world scale, NFSP learnt a strategy that approached the performance of state-of-the-art, superhuman algorithms based on significant domain expertise.
中文速览
在不依赖任何领域专家知识的前提下，如何让AI自主学习到多人不完全信息博弈（imperfect-information game）中的纳什均衡（Nash equilibrium）策略，是一个长期悬而未决的难题。作者提出了"神经虚拟自博弈"（Neural Fictitious Self-Play，NFSP），将虚拟博弈（fictitious play）这一经典博弈论方法与深度强化学习相结合：每个智能体同时维护两个神经网络，一个通过强化学习学习针对对手的最优反应，另一个通过监督学习拟合自身历史行为的平均策略，两者混合后指导实际决策。在小规模的Leduc扑克上，NFSP稳定收敛到纳什均衡，而普通强化学习方法则发散；在真实规模的"限注德州扑克"（Limit Texas Hold'em）上，NFSP仅凭原始输入就学到了接近顶尖超人类算法（后者依赖大量人工抽象与专家设计）的竞争性策略。这项工作的重要性在于，它首次提供了一条端到端、无需人工干预即可逼近纳什均衡的可扩展路径，为将博弈论求解方法推广到安全、金融、交通等复杂现实场景奠定了基础。
原文 arXiv:1603.01121;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1603.01121v2