Are We There Yet? A Decision Framework for Replacing Term-Based Retrieval with Dense Retrieval Systems
Sebastian Hofstätter TU Wien , Nick Craswell Microsoft , Bhaskar Mitra Microsoft , Hamed Zamani University of Massachusetts Amherst and Allan Hanbury TU Wien
Abstract
Recently, several dense retrieval (DR) models have demonstrated competitive performance to term-based retrieval that are ubiquitous in search systems. In contrast to term-based matching, DR projects queries and documents into a dense vector space and retrieves results via (approximate) nearest neighbor search. Deploying a new system, such as DR, inevitably involves tradeoffs in aspects of its performance. Established retrieval systems running at scale are usually well understood in terms of effectiveness and costs, such as query latency, indexing throughput, or storage requirements. In this work, we propose a framework with a set of criteria that go beyond simple effectiveness measures to thoroughly compare two retrieval systems with the explicit goal of assessing the readiness of one system to replace the other. This includes careful tradeoff considerations between effectiveness and various cost factors. Furthermore, we describe guardrail criteria, since even a system that is better on average may have systematic failures on a minority of queries. The guardrails check for failures on certain query characteristics and novel failure types that are only possible in dense retrieval sy
中文速览
稠密检索（Dense Retrieval，DR）近年来在学术榜单上表现抢眼，但学界缺乏一套系统方法来判断它究竟够不够格在真实搜索系统中取代经典的倒排索引检索。本文提出一个决策框架，除了比较平均检索效果，还引入了成本权衡（索引构建、查询延迟、存储等）以及"护栏"（guardrail）检验——专门检测模型在罕见词查询、长文档、分布外数据等场景下是否存在系统性失败，因为这些问题在平均指标中往往被掩盖。作者将该框架应用于网页检索场景，发现最先进的稠密检索模型在效果、鲁棒性、延迟等几乎所有维度上都通过了测试，唯一的硬伤是向量化（vectorization）的构建成本远高于传统索引，在硬件预算有限时这一点足以成为否决理由。这一框架的价值在于为工业界提供了一套理性、可复用的系统替换决策流程，而非仅仅比拼榜单分数。
原文 arXiv:2206.12993;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.12993v1