Dynamic Planning in Open-Ended Dialogue using Reinforcement Learning
Deborah Cohen, Moonkyung Ryu, Yinlam Chow, Orgad Keller, Ido Greenberg, Avinatan Hassidim, Michael Fink, Yossi Matias, Idan Szpektor, Craig Boutilier, Gal Elidan debbycohen,mkryu,yinlamchow,orgad,idogreenberg,avinatan,fink,yossi, Google Research
Abstract
Despite recent advances in natural language understanding and generation, and decades of research on the development of conversational bots, building automated agents that can carry on rich open-ended conversations with humans “in the wild” remains a formidable challenge. In this work we develop a real-time, open-ended dialogue system that uses reinforcement learning (RL) to power a bot’s conversational skill at scale. Our work pairs the succinct embedding of the conversation state generated using SOTA (supervised) language models with RL techniques that are particularly suited to a dynamic action space that changes as the conversation progresses. Trained using crowd-sourced data, our novel system is able to substantially exceeds the (strong) baseline supervised model with respect to several metrics of interest in a live experiment with real users of the Google Assistant.
中文速览
让对话机器人真正学会"聊天"而不只是逐轮回复，是这项工作的核心目标。研究者在谷歌助手（Google Assistant）的动物知识问答场景中，将强化学习（Reinforcement Learning, RL）与预训练语言模型（RNN、Transformer）相结合：用语言模型把对话历史压缩成紧凑的向量表示作为状态，再从内容提供器动态生成的少量候选话语中用RL选取并组合出最终回复，从而把对话规划和内容生成解耦开来。在真实用户的在线实验中，这套系统相比强基线监督模型，在对话轮数、用户积极反馈和合作性回复等多项指标上取得了显著提升。这一成果标志着RL首次被成功部署到大规模、实时、开放域对话系统中，为让智能助手真正具备跨轮次动态规划能力提供了可落地的路径。
原文 arXiv:2208.02294;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2208.02294v1