Hello, It’s GPT-2 - How Can I Help You? Towards the Use of Pretrained Language Models for Task-Oriented Dialogue Systems
Paweł Budzianowski1,2,3 Ivan Vulić2,3 1Engineering Department, Cambridge University, UK 2Language Technology Lab, Cambridge University, UK 3PolyAI Limited, London, UK
Abstract
Data scarcity is a long-standing and crucial challenge that hinders quick development of task-oriented dialogue systems across multiple domains: task-oriented dialogue models are expected to learn grammar, syntax, dialogue reasoning, decision making, and language generation from absurdly small amounts of task-specific data. In this paper, we demonstrate that recent progress in language modeling pre-training and transfer learning shows promise to overcome this problem. We propose a task-oriented dialogue model that operates solely on text input: it effectively bypasses explicit policy and language generation modules. Building on top of the TransferTransfo framework Wolf et al. (2019) and generative model pre-training Radford et al. (2019), we validate the approach on complex multi-domain task-oriented dialogues from the MultiWOZ dataset. Our automatic and human evaluations show that the proposed model is on par with a strong task-specific neural baseline. In the long run, our approach holds promise to mitigate the data scarcity problem, and to support the construction of more engaging and more eloquent task-oriented conversational agents.
中文速览
任务导向对话系统（task-oriented dialogue system）长期面临标注数据严重不足的瓶颈，训练一个能同时处理语言理解、对话管理和回复生成的系统需要大量领域专属数据。研究者将大规模预训练语言模型GPT/GPT-2与TransferTransfo框架引入这一领域，把信念状态（belief state）和知识库状态全部转换为纯文本格式直接输入模型，从而完全绕开传统的策略模块和自然语言生成模块，用一个统一的序列到序列架构端到端地完成对话。在包含7个领域、逾万段对话的MultiWOZ基准数据集上，自动评测显示GPT系列模型与专为任务设计的强基线相差不大，人工评测则显示两者无显著差异。这项工作的意义在于：它证明了通用预训练模型只需少量微调就能胜任复杂任务导向对话，为缓解数据稀缺问题、降低跨域构建对话系统的门槛提供了一条切实可行的路径。
原文 arXiv:1907.05774;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.05774v2