Language Instructed Reinforcement Learning for Human-AI Coordination
Hengyuan Hu Dorsa Sadigh
Abstract
One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL , that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance through human evaluations in Hanabi.
中文速览
多智能体强化学习（Multi-Agent RL）在协作游戏中往往会收敛到各种奇怪的均衡策略，而人类玩家根本不知道该怎么跟这些策略配合。为了解决这个问题，研究者提出了 instructRL 框架：让人类用自然语言（比如"选和Alice报的同一个数字"）告诉 AI 应该遵循什么策略，然后用大型语言模型（LLM）根据这条指令生成一个"先验策略"，再用这个先验去约束强化学习的优化目标，把 AI 引导到符合人类期望的均衡点上。在专门设计的 Say-Select 小游戏和著名的 Hanabi 协作牌类基准测试中，instructRL 都能按照指令收敛出直觉上自然、人类可理解的策略，而且人类玩家一旦知道 AI 在遵循哪条语言指令，人机配合得分就会显著提升。这项工作的重要性在于：它不需要任何人类行为数据，仅凭一句自然语言就能让 AI 和人类"说好怎么配合"，为无人类数据场景下的人机协作提供了一条切实可行的新路径。
原文 arXiv:2304.07297;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.07297v2