Collaborative Multi-Agent Dialogue Model Training Via Reinforcement Learning
Alexandros Papangelis, Yi-Chia Wang, Piero Molino, Gokhan Tur Uber AI San Francisco, California {apapangelis, yichia.wang, piero,
Abstract
We present the first complete attempt at concurrently training conversational agents that communicate only via self-generated language. Using DSTC2 as seed data, we trained natural language understanding (NLU) and generation (NLG) networks for each agent and let the agents interact online. We model the interaction as a stochastic collaborative game where each agent (player) has a role (“assistant”, “tourist”, “eater”, etc.) and their own objectives, and can only interact via natural language they generate. Each agent, therefore, needs to learn to operate optimally in an environment with multiple sources of uncertainty (its own NLU and NLG, the other agent’s NLU, Policy, and NLG). In our evaluation, we show that the stochastic-game agents outperform deep learning based supervised baselines.
中文速览
训练两个只靠自己生成的文字互相对话的智能体，一直是对话系统领域的难题——此前的研究要么让智能体用对话行为（dialogue act）而非真实语言交流，要么依赖固定的对手或全知的裁判。这项工作首次让两个角色不同（信息寻求者与信息提供者）的对话智能体完全通过各自神经网络生成的自然语言进行交互，把整个过程建模为随机博弈（stochastic game），并用专为非平稳环境设计的多智能体强化学习算法 WoLF-PHC 同步训练双方的对话策略。实验以餐厅预订数据集 DSTC2 为种子数据，结果表明多智能体随机博弈框架下训练出的智能体在对话质量上明显优于基于深度学习的单智能体监督学习基线。这一方案的意义在于：它大幅减少了对人工标注数据的依赖，并能通过智能体互动自动生成原始数据中不存在的对话行为，为低资源场景下构建高质量对话系统提供了新思路。
原文 arXiv:1907.05507;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.05507v2