Task-aware Retrieval with Instructions
Akari Asai†‡, Timo Schick†, Patrick Lewis†, Xilun Chen†, Gautier Izacard†♠, Sebastian Riedel♣, Hannaneh Hajishirzi‡♡, Wen-tau Yih† †Meta AI ‡University of Washington ♠ ENS, PSL University、Inria ♡Allen Institute for AI ♣University College London
Abstract
We study the problem of retrieval with instructions, where users of a retrieval system explicitly describe their intent along with their queries. We aim to develop a general-purpose task-aware retrieval system using multi-task instruction tuning, which can follow human-written instructions to find the best documents for a given query. We introduce the first large-scale collection of approximately 40 retrieval datasets with instructions, $\mathbb{BERRI}$ , and present $\mathbb{TART}$ , a multi-task retrieval system trained on $\mathbb{BERRI}$ with instructions. $\mathbb{TART}$ shows strong capabilities to adapt to a new retrieval task via instructions and advances the state of the art on two zero-shot retrieval benchmarks, BEIR and LOTTE, outperforming models up to three times larger. We further introduce a new evaluation setup, $\mathbb{X}^{2}$ -Retrieval to better reflect real-world scenarios, where diverse domains and tasks are pooled and a system needs to find documents aligning users’ intents. In this setup, $\mathbb{TART}$ significantly outperforms competitive baselines, further demonstrating the effectiveness of guiding retrieval with instructions.111Code, data and pretrained
中文速览
现有的检索系统往往只能针对单一任务训练，无法灵活应对用户多变的检索意图——比如同一个查询，有人想找答案段落，有人想找相似问题，有人想找代码片段。为此，本文提出"带指令的检索"（retrieval with instructions）新范式，让用户用自然语言明确描述检索意图，系统再据此找到最合适的文档。研究者构建了首个大规模检索指令数据集 BERRI（约40个多领域数据集，每个任务平均配备3.5条专家撰写的指令），并在此基础上训练了多任务检索模型 TART，其中 TART-full 采用交叉编码器架构，在零样本检索基准 BEIR 和 LOTTE 上超越了参数量多达三倍的竞争模型。此外，本文还提出了更贴近真实场景的跨任务跨领域评测设置 X²-Retrieval，TART 在该设置下同样大幅领先基线，证明了用自然语言指令引导检索这一思路的有效性与通用价值。
原文 arXiv:2211.09260;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.09260v2