Asking and Answering Questions to Evaluate the Factual Consistency of Summaries
Alex Wang111Work done while interning at Facebook. New York University、Kyunghyun Cho Facebook AI New York University、Mike Lewis Facebook AI
Abstract
Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS222Pronounced “kags”. that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.
中文速览
摘要式自动摘要（abstractive summarization）模型经常在生成的文本中引入事实错误，而现有的自动评估指标（如 ROUGE、BLEU）对这类错误几乎"视而不见"，因为它们只按 n-gram 重叠计分，忽视语义层面的差异。为此，作者提出了 QAGS（Question Answering and Generation for Summarization）这一自动评估方法：先基于生成的摘要自动生成一批问题，再分别用摘要和原文去回答这些问题，最后通过比较两组答案的相似度来判断摘要与原文在事实上是否一致——如果摘要忠实于原文，两者给出的答案应该相近。在 CNN/DailyMail 和 XSUM 两个数据集上，QAGS 与人工事实一致性判断的皮尔逊相关系数远高于所有对比指标（例如在 CNN/DM 上达到 54.52，而 ROUGE-2 仅为 17.72），且具备一定的可解释性——生成的问答对能直接指出摘要中哪些词句存在错误及原因。这项工作为自动检测生成文本中的事实错误提供了一个更可靠、更实用的评估工具，有助于推动摘要模型向更高事实准确性的方向改进。
原文 arXiv:2004.04228;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.04228v1