Chat as Expected: Learning to Manipulate Black-box Neural Dialogue Models
Haochen Liu Michigan State University Tyler Derr Michigan State University、Zhiwei Wang Michigan State University Jiliang Tang Michigan State University
Abstract
Recently, neural network based dialogue systems have become ubiquitous in our increasingly digitalized society. However, due to their inherent opaqueness, some recently raised concerns about using neural models are starting to be taken seriously. In fact, intentional or unintentional behaviors could lead to a dialogue system to generate inappropriate responses. Thus, in this paper, we investigate whether we can learn to craft input sentences that result in a black-box neural dialogue model being manipulated into having its outputs contain target words or match target sentences. We propose a reinforcement learning based model that can generate such desired inputs automatically. Extensive experiments on a popular well-trained state-of-the-art neural dialogue model show that our method can successfully seek out desired inputs that lead to the target outputs in a considerable portion of cases. Consequently, our work reveals the potential of neural dialogue models to be manipulated, which inspires and opens the door towards developing strategies to defend them.
中文速览
神经对话模型（neural dialogue model）正被广泛部署在聊天机器人等场景中，但它们是否会被人为操纵、输出特定目标内容，始终是个安全隐患。为了回答这个问题，研究者提出了一个基于强化学习（reinforcement learning）的框架TDGPN，专门用来自动生成能让黑盒对话模型"说出"目标词语或目标句子的输入文本——整个过程无需接触模型内部参数，也不依赖传统梯度优化，因为输入是离散的自然语言、梯度根本无法直接回传。实验在主流神经对话模型上验证了该方法的有效性，在相当比例的情况下成功诱导模型输出预设的目标内容。这项研究揭示了对话系统被恶意操控的现实风险，同时为后续研究如何防御此类攻击提供了重要参考。
原文 arXiv:2005.13170;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.13170v1