Detecting Hallucinated Content in Conditional Neural Sequence Generation
Chunting Zhou1, Graham Neubig1, Jiatao Gu2, Mona Diab2, Paco Guzman2, Luke Zettlemoyer2, Marjan Ghazvininejad2 Language Technologies Institute, Carnegie Mellon University1 Facebook AI Research2 Most work was done during an internship at FAIR.
Abstract
Neural sequence models can generate highly fluent sentences, but recent studies have also shown that they are also prone to hallucinate additional content not supported by the input. These variety of fluent but wrong outputs are particularly problematic, as it will not be possible for users to tell they are being presented incorrect content. To detect these errors, we propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) and collect new manually annotated evaluation sets for this task. We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data that includes automatically inserted hallucinations Experiments on machine translation (MT) and abstractive summarization demonstrate that our proposed approach consistently outperforms strong baselines on all benchmark datasets. We further demonstrate how to use the token-level hallucination labels to define a fine-grained loss over the target sequence in low-resource MT and achieve significant improvements over strong baseline methods.We also apply our method to word-level quality estimation for MT and show its effe
中文速览
神经序列模型（如机器翻译和自动摘要系统）能生成流畅的文本，却经常"无中生有"——输出内容在输入中根本找不到依据，这类"幻觉"（hallucination）问题因为表面流畅而极难被用户察觉。作者提出在词元（token）级别逐词判断模型输出是否属于幻觉的新任务，并构建了人工标注的评测数据集；核心方法是先用预训练去噪自编码器（BART）自动往正确译文里插入幻觉词，生成带标注的合成训练数据，再用此数据微调预训练跨语言或单语言模型，使其学会逐词标记幻觉。在机器翻译和抽象摘要的多个基准上，该方法在无需任何人工标注训练数据的情况下平均F1达到约0.6，持续优于各类强基线；进一步把词元级幻觉标签引入低资源机器翻译的损失函数设计，翻译质量也大幅提升。这项工作为量化和减少生成模型的不忠实输出提供了实用工具，对安全部署神经文本生成系统具有重要意义。
原文 arXiv:2011.02593;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.02593v3