QAMParI: A Benchmark for Open-domain Questions with Many Answers
Samuel Joseph Amouyal Tomer Wolfson Ohad Rubin Ori Yoran Jonathan Herzig Jonathan Berant Blavatnik School of Computer Science, Tel Aviv University, Israel {samuel.amouyal, ohad.rubin,
Abstract
Existing benchmarks for open-domain question answering (ODQA) typically focus on questions whose answers are all in a single paragraph. By contrast, many natural questions, such as “What players were drafted by the Brooklyn Nets?” have a long list of answers extracted from multiple paragraphs. Answering such questions requires retrieving and reading many passages from a large corpus. We introduce QAMParI, an ODQA benchmark, where answers are lists of entities, spread across many paragraphs. We created QAMParI by (a) generating questions with multiple answers from Wikipedia’s knowledge graph and tables, (b) automatically pairing answers with supporting evidence in Wikipedia paragraphs, and (c) manually paraphrasing questions and validating each answer. Across a wide range of ODQA models, we find that QAMParI is challenging in terms of both passage retrieval and answer generation, with models reaching an F1 score of 32.8 at best. We view QAMParI as a valuable resource for ODQA research, which will aid to develop models that handle a broad range of question types, including single and multi-answer questions.
中文速览
开放域问答（ODQA）领域长期聚焦于"单一答案"类问题，而现实中大量问题（如"哪些球员曾被布鲁克林网队选秀？"）需要从海量段落中汇聚十几甚至更多个答案，现有模型和评测基准对此几乎无能为力。为此，作者构建了QAMParI基准——通过挖掘维基百科知识图谱（Wikidata）和表格自动生成多答案问题，再用自然语言推理模型验证每个答案的段落证据，最后通过众包人工改写与校验，形成涵盖简单、组合、交集三类问题共6万余条训练样本和2000条开发/测试样本。在这一基准上，BM25/DPR检索器搭配RAG或FiD阅读器等主流模型的最高F1分仅为32.8，且仅在31.2%的测试样例上能召回80%以上的正确答案，暴露出现有系统在多段落检索与多答案生成上的严重短板。QAMParI的价值在于推动社区建立能同时应对单答案与多答案问题的通用ODQA评测体系，而非仅在单一答案数据集上刷榜。
原文 arXiv:2205.12665;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.12665v4