RealTime QA: What’s the Answer Right Now?
Jungo Kasai♡♡{}^{\heartsuit}start_FLOATSUPERSCRIPT ♡ end_FLOATSUPERSCRIPT Keisuke Sakaguchi♣□normal-♣normal-□{}^{\clubsuit\square}start_FLOATSUPERSCRIPT ♣ □ end_FLOATSUPERSCRIPT Yoichi Takahashi♣normal-♣{}^{\clubsuit}start_FLOATSUPERSCRIPT ♣ end_FLOATSUPERSCRIPT Ronan Le Bras♢normal-♢{}^{\diamondsuit}start_FLOATSUPERSCRIPT ♢ end_FLOATSUPERSCRIPT Akari Asai♠normal-♠{}^{\spadesuit}start_FLOATSUPERSCRIPT ♠ end_FLOATSUPERSCRIPT Xinyan Velocity Yu\varheartsuit\varheartsuit{}^{\varheartsuit}start_FLOATSUPERSCRIPT end_FLOATSUPERSCRIPT Dragomir Radev\vardiamondsuit\vardiamondsuit{}^{\vardiamondsuit}start_FLOATSUPERSCRIPT end_FLOATSUPERSCRIPT Noah A. Smith♢♠normal-♢normal-♠{}^{\diamondsuit\spadesuit}start_FLOATSUPERSCRIPT ♢ ♠ end_FLOATSUPERSCRIPT Yejin Choi♢♠normal-♢normal-♠{}^{\diamondsuit\spadesuit}start_FLOATSUPERSCRIPT ♢ ♠ end_FLOATSUPERSCRIPT Kentaro Inui△♣□normal-△normal-♣normal-□{}^{\triangle\clubsuit\square}start_FLOATSUPERSCRIPT △ ♣ □ end_FLOATSUPERSCRIPT ♡♡{}^{\heartsuit}start_FLOATSUPERSCRIPT ♡ end_FLOATSUPERSCRIPTToyota Technological Institute at Chicago ♣♣{}^{\clubsuit}start_FLOATSUPERSCRIPT ♣ end_FLOATSUPERSCRIPTTohoku University □□{}^{\square}start_FLOATSUPERSCRIPT □ end_FLOATSUPERSCRIPTRIKEN ♢♢{}^{\diamondsuit}start_FLOATSUPERSCRIPT ♢ end_FLOATSUPERSCRIPTAllen Institute for AI ♠♠{}^{\spadesuit}start_FLOATSUPERSCRIPT ♠ end_FLOATSUPERSCRIPTUniversity of Washington \varheartsuit\varheartsuit{}^{\varheartsuit}start_FLOATSUPERSCRIPT end_FLOATSUPERSCRIPTUniversity of Southern California \vardiamondsuit\vardiamondsuit{}^{\vardiamondsuit}start_FLOATSUPERSCRIPT end_FLOATSUPERSCRIPTYale University △△{}^{\triangle}start_FLOATSUPERSCRIPT △ end_FLOATSUPERSCRIPTMBZUAI @realtimeqa Work was done during JK’s internship at AI2.
Abstract
We introduce RealTime QA, a dynamic question answering (QA) platform that announces questions and evaluates systems on a regular basis (weekly in this version). RealTime QA inquires about the current world, and QA systems need to answer questions about novel events or information. It therefore challenges static, conventional assumptions in open-domain QA datasets and pursues instantaneous applications. We build strong baseline models upon large pretrained language models, including GPT-3 and T5. Our benchmark is an ongoing effort, and this paper presents real-time evaluation results over the past year. Our experimental results show that GPT-3 can often properly update its generation results, based on newly-retrieved documents, highlighting the importance of up-to-date information retrieval. Nonetheless, we find that GPT-3 tends to return outdated answers when retrieved documents do not provide sufficient information to find an answer. This suggests an important avenue for future research: can an open-domain QA system identify such unanswerable cases and communicate with the user or even the retrieval module to modify the retrieval results? We hope that RealTime QA will spur progres
中文速览
现实世界每周都在发生新事件，而现有的问答系统评测基准几乎都只针对静态知识，完全跟不上"当下"的信息需求——RealTime QA 正是为了填补这一空缺而建立的动态问答平台，每周自动从CNN、USA Today等新闻网站抓取多项选择题并向外发布，要求参赛系统在当周提交答案。研究者以GPT-3和T5为基础构建了开卷（open-book，配合实时检索）与闭卷（closed-book，只依赖预训练知识）两类基线系统，并在超过一年的时间里持续积累了1,470道评测题目。实验结果显示，结合实时新闻检索的GPT-3大幅优于仅用旧版维基百科的模型或纯闭卷模型，证明大型语言模型确实能够利用新检索到的文档来更新答案；但当检索结果信息不足时，模型仍会给出过时的回答，暴露出实时检索准确性的明显短板。这项工作不仅为问答研究提供了持续演化的真实测试床，也为未来探索"模型如何识别无法回答的问题并主动改进检索"这一方向指明了道路。
原文 arXiv:2207.13332;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2207.13332v2