CQSumDP: A ChatGPT-Annotated Resource for Query-Focused Abstractive Summarization Based on Debatepedia
Md Tahmid Rahman Laskar1,3, Mizanur Rahman2,3, Israt Jahan3, Enamul Hoque3, Jimmy Huang3, 1Dialpad Canada Inc., 2Royal Bank of Canada, 3York University Toronto, Ontario, Canada Contact Author. All work being done at York University.
Abstract
Debatepedia is a publicly available dataset consisting of arguments and counter-arguments on controversial topics that has been widely used for the single-document query-focused abstractive summarization task in recent years. However, it has been recently found that this dataset is limited by noise and even most queries in this dataset do not have any relevance to the respective document. In this paper, we present a methodology for cleaning the Debatepedia dataset by leveraging the generative power of large language models to make it suitable for query-focused abstractive summarization. More specifically, we harness the language generation capabilities of ChatGPT to regenerate its queries. We evaluate the effectiveness of the proposed ChatGPT annotated version of the Debatepedia dataset using several benchmark summarization models and demonstrate that the newly annotated version of Debatepedia outperforms the original dataset in terms of both query relevance as well as summary generation quality. We will make this annotated and cleaned version of the dataset publicly available.
中文速览
现有的 Debatepedia 数据集虽然被广泛用于"以问题为中心的摘要生成"（query-focused abstractive summarization）任务，却存在严重质量问题——超过半数的问题（query）与对应文档毫无关联，还有大量单词摘要和封闭式是非题，导致模型根本无法真正学到"围绕特定问题生成摘要"的能力。为此，作者提出用 ChatGPT 作为自动标注工具，先按文档长度和摘要长度过滤掉噪声样本，再让 ChatGPT 根据文档-摘要对重新生成更相关的问题，从而得到一个更干净的新版本数据集 CQSumDP。在多个主流摘要模型上的实验表明，用新数据集训练和评测的模型，无论是问题相关性还是摘要质量（ROUGE 等指标）都明显优于原始数据集。这项工作不仅为争议性文本的查询摘要研究提供了可靠的开源数据资源，也为用大语言模型低成本替代人工标注提供了一条可复用的方法路径。
原文 arXiv:2305.06147;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.06147v1