CiteBench: A Benchmark for Scientific Citation Text Generation
Martin Funkquist1 , Ilia Kuznetsov2, Yufang Hou3, Iryna Gurevych2 1Linköping University 2UKP Lab, Department of Computer Science and Hessian Center for AI (hessian.AI) Technical University of Darmstadt 3IBM Research Europe - Ireland Work done during an internship at UKP Lab.
Abstract
Science progresses by building upon the prior body of knowledge documented in scientific publications. The acceleration of research makes it hard to stay up-to-date with the recent developments and to summarize the ever-growing body of prior work. To address this, the task of citation text generation aims to produce accurate textual summaries given a set of papers-to-cite and the citing paper context. Due to otherwise rare explicit anchoring of cited documents in the citing paper, citation text generation provides an excellent opportunity to study how humans aggregate and synthesize textual knowledge from sources. Yet, existing studies are based upon widely diverging task definitions, which makes it hard to study this task systematically. To address this challenge, we propose CiteBench: a benchmark for citation text generation that unifies multiple diverse datasets and enables standardized evaluation of citation text generation models across task designs and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transferability between the datasets, and deliver new insights into the task definition and evaluation to guide future re
中文速览
科学文献中的"引用文本生成"（citation text generation）任务，要求模型根据被引论文内容和引用方的上下文，自动撰写出恰当的引用描述句或相关工作段落，但该领域长期存在任务定义五花八门、数据集互不兼容、评估标准各行其是的问题，导致不同方法之间难以公平比较。为此，作者构建了CiteBench这一统一基准，将四个来自不同领域、任务设计各异的数据集整合到同一形式化框架下，并配套提供了标准化评估工具包——除了ROUGE和BERTScore等常规指标外，还引入了引用意图分类和CORWA话语结构标注两种定性分析手段。在此基准上，研究者系统测试了多种强基线模型（包括无监督抽取式方法和基于Longformer的生成式模型），发现不同数据集之间的模型迁移性普遍有限，且现有定量指标难以全面反映生成质量。CiteBench为引用文本生成研究提供了首个公平可复现的测试平台，有助于推动该领域从碎片化走向系统化研究。
原文 arXiv:2212.09577;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.09577v3