Detecting and Classifying Malevolent Dialogue Responses: Taxonomy, Data and Methodology
Yangjun Zhang University of Amsterdam, Amsterdam, The Netherlands , Pengjie Ren University of Amsterdam, Amsterdam, The Netherlands and Maarten de Rijke University of Amsterdam, Amsterdam, The Netherlands、Ahold Delhaize, Zaandam, The Netherlands
Abstract
Conversational interfaces are increasingly popular as a way of connecting people to information. Corpus-based conversational interfaces are able to generate more diverse and natural responses than template-based or retrieval-based agents. With their increased generative capacity of corpus-based conversational agents comes the need to classify and filter out malevolent responses that are inappropriate in terms of content and dialogue acts. Previous studies on the topic of recognizing and classifying inappropriate content are mostly focused on a certain category of malevolence or on single sentences instead of an entire dialogue. In this paper, we define the task of Malevolent Dialogue Response Detection and Classification (MDRDC). We make three contributions to advance research on this task. First, we present a Hierarchical Malevolent Dialogue Taxonomy (HMDT). Second, we create a labelled multi-turn dialogue dataset and formulate the MDRDC task as a hierarchical classification task over this taxonomy. Third, we apply state-of-the-art text classification methods to the MDRDC task and report on extensive experiments aimed at assessing the performance of these approaches.
中文速览
对话系统生成的回复有时会包含仇恨、侮辱、威胁或不道德内容，却一直缺乏系统性的检测框架和对话级别的数据资源。作者为此提出了"恶意对话回复检测与分类（MDRDC）"任务，并从心理学、伦理学等多学科文献中归纳出一套三层"层级恶意对话分类体系（HMDT）"，涵盖负面情绪、不当行为和伦理失范三大维度。在此基础上，他们从 Twitter 收集多轮对话并通过众包标注，构建了首个面向多轮对话的恶意回复标注数据集，同时让标注者对恶意回复进行改写以增加数据多样性。用 CNN、RNN、GCN 和 BERT 等主流文本分类方法在该数据集上进行系统评测后发现，引入对话上下文和改写样本能显著提升分类效果，但整体性能仍远未达到实际部署所需水平，说明构建安全可信的对话系统仍面临巨大挑战。这项工作填补了恶意对话内容检测领域在分类体系、多轮数据集和基准评测三方面的空白，为后续研究提供了可复用的资源和明确的改进方向。
原文 arXiv:2008.09706;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2008.09706v1