WiCE: Real-World Entailment for Claims in Wikipedia
Ryo Kamoi♢ Tanya Goyal♢ Juan Diego Rodriguez♠♢ Greg Durrett♢ ♢ Department of Computer Science, The University of Texas at Austin ♠ Applied Research Laboratories, The University of Texas at Austin
Abstract
Textual entailment models are increasingly applied in settings like fact-checking, presupposition verification in question answering, or summary evaluation. However, these represent a significant domain shift from existing entailment datasets, and models underperform as a result. We propose WiCE, a new fine-grained textual entailment dataset built on natural claim and evidence pairs extracted from Wikipedia. In addition to standard claim-level entailment, WiCE provides entailment judgments over sub-sentence units of the claim, and a minimal subset of evidence sentences that support each subclaim. To support this, we propose an automatic claim decomposition strategy using GPT-3.5 which we show is also effective at improving entailment models’ performance on multiple datasets at test time. Finally, we show that real claims in our dataset involve challenging verification and retrieval problems that existing models fail to address.111Our data is available at: https://github.com/ryokamoi/wice
中文速览
现实世界中的事实核查、问答验证、摘要评估等任务正在大量依赖文本蕴含（textual entailment）模型，但现有数据集都是短句对、合成负例，根本撑不住这些实际场景。为此，研究者构建了 WiCE（Wikipedia Citation Entailment）数据集：从维基百科真实引用链中提取声明与证据对，并借助 GPT-3.5 的少样本提示自动将复杂声明拆解为若干子声明（subclaim），再由众包标注员对每条子声明分别标注"是否被支持""哪些证据句支撑它""哪些词语不被支持"，最终得到约 5900 条子声明级和 2000 条声明级的细粒度蕴含样本。实验表明，现有 NLI 模型在该数据集上表现明显低于人类水平，检索性能尤其薄弱，而利用同样的声明拆解策略（Claim-Split）对输入进行预处理，能在多个数据集上提升现成模型的分类表现。这项工作填补了 NLI 研究与真实应用场景之间的鸿沟，为更细粒度、更贴近实战的事实核查系统提供了重要的数据基础和方法参考。
原文 arXiv:2303.01432;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.01432v2