Continuously Learning Neural Dialogue Management
Pei-Hao Su, Milica Gašić, Nikola Mrkšić, Lina Rojas-Barahona, Stefan Ultes, David Vandyke, Tsung-Hsien Wen and Steve Young Department of Engineering, University of Cambridge, Cambridge, UK {phs26, mg436, nm480, lmr46, su259, djv27, thw28,
Abstract
We describe a two-step approach for dialogue management in task-oriented spoken dialogue systems. A unified neural network framework is proposed to enable the system to first learn by supervision from a set of dialogue data and then continuously improve its behaviour via reinforcement learning, all using gradient-based algorithms on one single model. The experiments demonstrate the supervised model’s effectiveness in the corpus-based evaluation, with user simulation, and with paid human subjects. The use of reinforcement learning further improves the model’s performance in both interactive settings, especially under higher-noise conditions.
中文速览
任务型口语对话系统（Spoken Dialogue System）的对话管理模块既要稳定可靠，又要能适应真实环境中不断变化的噪声条件，而单纯依赖监督学习或强化学习各有明显短板。研究者提出了一个两阶段统一神经网络框架：第一阶段用人工收集的对话语料做监督学习，让模型快速习得基本响应策略；第二阶段在不改变网络结构的前提下，直接用策略梯度强化学习（policy-gradient reinforcement learning）对同一个模型进行在线微调，使其在与用户的真实交互中持续改进。实验结果显示，仅用约720条对话完成监督预训练后，模型就能取得较好的任务完成率，而在模拟用户和真实众包用户两种场景下，追加强化学习训练后成功率进一步提升，在高噪声条件下提升幅度尤为明显。这项工作的意义在于，它用一套轻量级的端到端可微框架，填平了监督学习与强化学习之间的鸿沟，为实际部署中低成本、快速迭代对话策略提供了切实可行的路径。
原文 arXiv:1606.02689;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1606.02689v1