High-Level Strategy Selection under Partial Observability in StarCraft: Brood War
Jonas Gehring, Da Ju, Vegard Mella, Daniel Gant, Nicolas Usunier, Gabriel Synnaeve Facebook AI Research
Abstract
We consider the problem of high-level strategy selection in the adversarial setting of real-time strategy games from a reinforcement learning perspective, where taking an action corresponds to switching to the respective strategy. Here, a good strategy successfully counters the opponent’s current and possible future strategies which can only be estimated using partial observations. We investigate whether we can utilize the full game state information during training time (in the form of an auxiliary prediction task) to increase performance. Experiments carried out within a StarCraft®: Brood War®111StarCraft is a trademark or registered trademark of Blizzard Entertainment, Inc., in the U.S. and/or other countries. Nothing in this paper should be construed as approval, endorsement, or sponsorship by Blizzard Entertainment, Inc. bot against strong community bots show substantial win rate improvements over a fixed-strategy baseline and encouraging results when learning with the auxiliary task.
中文速览
在星际争霸：母巢之战这类即时战略游戏里，如何根据对手的策略实时切换己方战术（build order）是一个关键难题，难点在于"战争迷雾"导致玩家只能看到部分战场信息。研究者将这个问题建模为强化学习中的Q值估计任务，让智能体在25个预设战术之间动态选择，并额外引入了一个辅助预测任务——在训练时利用全局完整信息来预测对手的兵种数量，以帮助网络学到更好的对手状态表示。实验在CherryPi机器人平台上对阵多个强力社区机器人，结果显示与固定战术基线相比，所有变体的胜率均有显著提升；加入辅助任务后Q值预测误差明显下降，胜率也有所改善，且模型能在比赛中输出合理的对手兵力预测。这项工作表明，把训练时可获取的隐藏状态信息以辅助任务的形式融入强化学习，是一种在不完全信息对抗场景下提升战略决策能力的有效途径。
原文 arXiv:1811.08568;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.08568v1