Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game
Zelai Xu Chao Yu Fei Fang Yu Wang Yi Wu
Abstract
Agents built with large language models (LLMs) have shown great potential across a wide range of domains. However, in complex decision-making tasks, pure LLM-based agents tend to exhibit intrinsic bias in their choice of actions, which is inherited from the model’s training data and results in suboptimal performance. To develop strategic language agents, i.e., agents that generate flexible language actions and possess strong decision-making abilities, we propose a novel framework that powers LLM-based agents with reinforcement learning (RL). We consider Werewolf, a popular social deduction game, as a challenging testbed that emphasizes versatile communication and strategic gameplay. To mitigate the intrinsic bias in language actions, our agents use an LLM to perform deductive reasoning and generate a diverse set of action candidates. Then an RL policy trained to optimize the decision-making ability chooses an action from the candidates to play in the game. Extensive experiments show that our agents overcome the intrinsic bias and outperform existing LLM-based agents in the Werewolf game. We also conduct human-agent experiments and find that our agents achieve human-level performanc
中文速览
大型语言模型（LLM）驱动的智能体在决策类游戏中存在一个棘手问题：即便模型推理正确，它的实际行动选择仍会受训练数据影响而产生固定偏好（intrinsic bias），就像让 GPT-4 玩"石头剪子布"时它会不成比例地偏向出石头一样。为此，研究者以狼人杀游戏为测试场景，提出了一套将 LLM 与强化学习（RL）结合的新框架：先用 LLM 进行演绎推理、推断玩家隐藏身份，并生成多个候选行动以打散固有偏好，再训练一个 RL 策略从这些候选中选出最优行动，同时通过与多样化智能体群体对战来提升策略鲁棒性。实验表明，该框架在循环赛制中胜率超越所有纯 LLM 基线，并在与真实人类对局中达到平均人类水平的表现。这一工作的意义在于，它为需要自由语言交互的开放行动空间提供了一条可行路径，不再局限于预定义的有限动作集，对构建兼具语言能力与战略决策能力的通用智能体具有重要参考价值。
原文 arXiv:2310.18940;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.18940v4