Towards End-to-End Learning for Dialog State Tracking and Management using Deep Reinforcement Learning
Tiancheng Zhao and Maxine Eskenazi Language Technologies Institute Carnegie Mellon University {tianchez,
Abstract
This paper presents an end-to-end framework for task-oriented dialog systems using a variant of Deep Recurrent Q-Networks (DRQN). The model is able to interface with a relational database and jointly learn policies for both language understanding and dialog strategy. Moreover, we propose a hybrid algorithm that combines the strength of reinforcement learning and supervised learning to achieve faster learning speed. We evaluated the proposed model on a 20 Question Game conversational game simulator. Results show that the proposed method outperforms the modular-based baseline and learns a distributed representation of the latent dialog state.
中文速览
传统的对话系统把自然语言理解、对话状态追踪、对话策略等模块分开训练，一个模块出错会连累后续所有模块，而且整体优化极为困难。这篇论文提出了一个基于深度循环Q网络（Deep Recurrent Q-Network, DRQN）的端到端框架，把上述三个模块合并成一个统一模型，让它同时学会理解用户语言、维护对话状态，并制定达成任务目标的对话策略，还能直接与关系型数据库交互。为了加快学习速度，作者进一步设计了一种混合算法，将强化学习与监督学习结合——在强化学习的长期奖励之外引入状态追踪标注作为短期监督信号，并通过生成合成经验来扩充训练数据。在一个需要语言理解与策略规划的"二十问游戏"模拟器上，该框架的表现超越了传统模块化基线，并学到了有意义的对话状态隐层表示，证明端到端的强化学习方案在任务型对话中具有切实可行的优势。
原文 arXiv:1606.02560;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.02560v2