Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
Bernd Bohnet∗ Vinh Q. Tran∗ Pat Verga∗ \ANDRoee Aharoni Daniel Andor Livio Baldini Soares Massimiliano Ciaramita Jacob Eisenstein Kuzman Ganchev Jonathan Herzig Kai Hui Tom Kwiatkowski Ji Ma Jianmo Ni Lierni Sestorain Saralegui Tal Schuster \ANDWilliam W. Cohen Michael Collins Dipanjan Das Donald Metzler Slav Petrov Kellie Webster† Google Research
Abstract
Large language models (LLMs) have shown impressive results while requiring little or no direct supervision. Further, there is mounting evidence that LLMs may have potential in information-seeking scenarios. We believe the ability of an LLM to attribute the text that it generates is likely to be crucial in this setting. We formulate and study Attributed QA as a key first step in the development of attributed LLMs. We propose a reproducible evaluation framework for the task and benchmark a broad set of architectures. We take human annotations as a gold standard and show that a correlated automatic metric is suitable for development.111We publicly release all system responses and their human and automatic ratings, at https://github.com/google-research-datasets/Attributed-QA Our experimental work gives concrete answers to two key questions (How to measure attribution?, and How well do current state-of-the-art methods perform on attribution?), and give some hints as to how to address a third (How to build LLMs with attribution?).
中文速览
大型语言模型（LLM）在问答等信息检索场景中表现出色，但它们生成的内容往往缺乏可追溯的依据，用户无法判断答案是否有据可查。为此，研究者提出了"带归因的问答"（Attributed QA）任务——要求系统不仅给出答案，还必须从固定语料库中指出支撑该答案的具体段落。他们设计了一套可复现的评测框架，以人工标注为金标准，并验证了一种基于自然语言推理的自动指标（AutoAIS）与人工评分高度相关，可用于日常开发迭代；同时对"先检索再阅读"、事后归因、端到端等多类主流架构进行了系统性对比测评。实验表明，当前最优方法在归因准确率上仍有相当大的提升空间，该工作为如何衡量归因质量、现有方法的真实水平以及未来如何构建具备归因能力的 LLM 提供了清晰的基准与方向。
原文 arXiv:2212.08037;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.08037v2