Shallow pooling for sparse labels
Negar Arabzadeh Alexandra Vtyurina Xinyi Yan Charles L. A. Clarke
Abstract
Recent years have seen enormous gains in core information retrieval tasks, including document and passage ranking. Datasets and leaderboards, and in particular the MS MARCO datasets, illustrate the dramatic improvements achieved by modern neural rankers. When compared with traditional information retrieval test collections, such as those developed by TREC, the MS MARCO datasets employ substantially more queries — thousands vs. dozens – with substantially fewer known relevant items per query — often just one. For example, 94% of the nearly seven thousand queries in the MS MARCO passage ranking development set have only a single known relevant passage, and no query has more than four. Given the sparsity of these relevance labels, the MS MARCO leaderboards track improvements with mean reciprocal rank (MRR). In essence, the known relevant item is treated as the “right answer” or “best answer”, with rankers scored on their ability to place this item as high in the ranking as possible. In working with these sparse labels, we have observed that the top items returned by a ranker often appear superior to judged relevant items. Others have reported the same observation.
中文速览
信息检索领域的排名系统近年来突飞猛进，但衡量进步的基准数据集MS MARCO存在一个根本性缺陷：它为每个查询平均只标注了一个"参考答案"，而这个答案未必是最佳答案，只是当年标注者碰巧选中的那个。研究者通过众包偏好判断实验发现，现代神经排序模型（如RocketQA）返回的最高排名结果，往往比官方标注答案更受用户青睐——这意味着即便一个假想的"完美排序器"把所有官方答案都排到第一位，它的实际表现仍不如当前最先进的模型，出现了"比完美还好"的荒谬局面。进一步对排行榜顶尖系统进行500条查询的结果池化和重新评估后，研究者确认：现有MS MARCO标注已无法可靠区分各排序系统之间的真实优劣差异，排行榜上的名次变动可能并不反映真正的技术进步。为此，作者建议未来的稀疏标注数据集应将唯一标注位置留给"当前已知最佳答案"，并通过持续的偏好判断机制动态更新，从而确保评测体系能真正识别检索质量的提升。
原文 arXiv:2109.00062;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.00062v2