Iterative Policy Learning in End-to-End Trainable Task-Oriented Neural Dialog Models
Abstract
In this paper, we present a deep reinforcement learning (RL) framework for iterative dialog policy optimization in end-to-end task-oriented dialog systems. Popular approaches in learning dialog policy with RL include letting a dialog agent to learn against a user simulator. Building a reliable user simulator, however, is not trivial, often as difficult as building a good dialog agent. We address this challenge by jointly optimizing the dialog agent and the user simulator with deep RL by simulating dialogs between the two agents. We first bootstrap a basic dialog agent and a basic user simulator by learning directly from dialog corpora with supervised training. We then improve them further by letting the two agents to conduct task-oriented dialogs and iteratively optimizing their policies with deep RL. Both the dialog agent and the user simulator are designed with neural network models that can be trained end-to-end. Our experiment results show that the proposed method leads to promising improvements on task success rate and total task reward comparing to supervised training and single-agent RL training baseline models.
中文速览
任务导向对话系统（task-oriented dialog system）长期依赖人工设计的用户模拟器来训练对话智能体，而构建一个高质量的用户模拟器本身就和构建对话智能体一样困难。为解决这一问题，研究者提出了一个端到端的深度强化学习（deep reinforcement learning）框架：先用真实对话语料对对话智能体和用户模拟器分别做监督预训练，再让两个神经网络智能体相互对话，通过任务完成情况给予奖励，迭代地同步优化双方的对话策略。实验表明，与纯监督训练和只优化单个智能体的强化学习基线相比，该方法在任务成功率和总任务奖励上均有显著提升。这项工作的意义在于：它摆脱了对人工规则用户模拟器的依赖，让两个可端到端训练的神经网络模型在相互博弈中共同进步，为构建更鲁棒、更易扩展的对话系统提供了新思路。
原文 arXiv:1709.06136;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1709.06136v1