Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
Bing Liu, Ian Lane
Abstract
In this paper, we present a deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems. Popular approaches in learning dialog policy with RL include letting a dialog agent to learn against a user simulator. Building a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. We address this challenge by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. We first bootstrap a basic dialog agent and a basic user simulator by learning directly from dialog corpora with supervised training. We then improve them further by letting the two agents to conduct task-oriented dialogs and iteratively optimizing their policies with deep RL. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Our experiment results show that the proposed method leads to promising improvements on task success rate and total task reward comparing to supervised training and single-agent RL training baseline models.
中文速览
任务型对话系统要在多轮交流中帮用户完成订餐、查信息等目标，但强化学习通常依赖可靠的用户模拟器，而设计这样的模拟器并不比设计对话系统容易。研究先用真实对话语料分别监督训练一个基础对话代理和用户模拟器，再让两者围绕随机用户目标进行对话，并用任务完成度产生的奖励反复联合优化双方策略；两者都采用可端到端训练的神经网络。实验显示，这种方法相比单纯监督学习和只训练对话代理的强化学习基线，在任务成功率和总奖励上都有明显提升。它的重要性在于不再依赖大量手工规则，就能同时提升代理和模拟器，为更灵活、能应对多样场景的任务型对话系统提供了数据驱动的训练路径。
原文 arXiv:1709.06136;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.06136v1