Language Instructed Reinforcement Learning for Human-AI Coordination
Hengyuan Hu Dorsa Sadigh
Abstract
One of the fundamental quests of AI is to produce agents that coordinate well with humans. This problem is challenging, especially in domains that lack high quality human behavioral data, because multi-agent reinforcement learning (RL) often converges to different equilibria from the ones that humans prefer. We propose a novel framework, instructRL , that enables humans to specify what kind of strategies they expect from their AI partners through natural language instructions. We use pretrained large language models to generate a prior policy conditioned on the human instruction and use the prior to regularize the RL objective. This leads to the RL agent converging to equilibria that are aligned with human preferences. We show that instructRL converges to human-like policies that satisfy the given instructions in a proof-of-concept environment as well as the challenging Hanabi benchmark. Finally, we show that knowing the language instruction significantly boosts human-AI coordination performance through human evaluations in Hanabi.
中文速览
多智能体强化学习在缺少高质量人类行为数据时，常会学到虽然得分高、却不符合人类习惯的协作方式，导致人与AI难以配合。instructRL让人用自然语言说明希望AI遵循的策略，再借助大语言模型根据指令生成行动偏好，并把它作为强化学习的参考，推动训练结果收敛到符合人类期待的均衡。实验表明，它在设计的Say-Select游戏和Hanabi基准上都能学出遵守指令、表现强且风格不同的策略，而且人类知道指令后，与AI的协作效果显著提升。这个方法不依赖大规模人类示范数据，却能用简单的语言约定减少协作中的“猜心”和试错，为训练更容易理解、配合的人机协作智能体提供了实用思路。
原文 arXiv:2304.07297;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.07297v2