Overview of the TREC 2021 deep learning track
Nick Craswell Microsoft {nickcr, Bhaskar Mitra Microsoft {nickcr, Emine Yilmaz University College London Daniel Campos University of Illinois Urbana-Champaign Neural Magic Inc Jimmy Lin University of Waterloo
Abstract
This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we refreshed both the document and the passage collections which also led to a nearly four times increase in the document collection size and nearly $16$ times increase in the size of the passage collection. Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year. We also found that single stage retrieval can achieve good performance on both tasks although they still do not perform at par with multistage retrieval pipelines. Finally, the increase in the collection size and the general data refresh raised some questions about completeness of NIST judgments and the quality of the training labels that were mapped to the new collections from the old ones which we discuss in this report.
中文速览
大规模训练数据真的能让神经检索系统更强吗？TREC 2021深度学习赛道（Deep Learning Track）用一套扩容后的新基准数据集MS MARCO v2来回答这个问题——文档集扩大了近4倍至约1190万篇，段落集扩大了近16倍至约1.38亿条，并修复了旧版本的字符编码和段落-文档映射等问题。实验结果表明，采用大规模预训练的深度神经排序模型依然全面超越BM25等传统检索方法；以密集检索（dense retrieval）为代表的单阶段检索流程表现已相当不错，但整体上仍逊色于"先粗检索再精排"的多阶段流水线。此外，数据集规模的急剧扩大也暴露出NIST相关性标注覆盖不完整、旧标注映射到新集合后质量下降等现实挑战，为后续大规模信息检索基准的构建敲响了警钟。
原文 arXiv:2507.08191;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2507.08191v1