TimelineQA: A Benchmark for Question Answering over Timelines
Wang-Chiew Tan, Jane Dwivedi-Yu, Yuliang Li, Lambert Mathias, Marzieh Saeidi*, Jing Nathan Yan+, and Alon Y. Halevy Meta Cornell University+
Abstract
Lifelogs are descriptions of experiences that a person had during their life. Lifelogs are created by fusing data from the multitude of digital services, such as online photos, maps, shopping and content streaming services. Question answering over lifelogs can offer personal assistants a critical resource when they try to provide advice in context. However, obtaining answers to questions over lifelogs is beyond the current state of the art of question answering techniques for a variety of reasons, the most pronounced of which is that lifelogs combine free text with some degree of structure such as temporal and geographical information.
中文速览
人们每天与各类数字服务产生的数据可以被整合成"生活日志"（lifelog），记录下一个人的经历，而让AI助手能够准确回答基于生活日志的问题，是构建真正实用的个人助理的关键一步。为此，作者构建并公开发布了TimelineQA——一个专门用于评测生活日志问答系统的基准数据集，它能自动为虚构人物生成包含时间、地点和各类活动的生活日志，并配套生成原子型（单跳）和复杂型（多跳、聚合）问答对。实验表明，对于简单的原子问题，抽取式问答系统显著优于当前主流的检索增强生成（RAG）系统；而对于需要跨多条记录做聚合推理的复杂问题，最好成绩由表格问答模型Tapex取得，但即便在已知相关记录子集的理想条件下，准确率也只有59%。这一结果说明现有顶尖问答技术在面对兼具自由文本与时空结构的生活日志时仍有巨大差距，TimelineQA的发布为推动该领域研究提供了标准化的评测平台。
原文 arXiv:2306.01069;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.01069v1