Overview of the TREC 2020 deep learning track
Nick Craswell Microsoft AI、Research {nickcr, Bhaskar Mitra Microsoft AI、Research {nickcr, University College London Emine Yilmaz University College London Daniel Campos University of Illinois Urbana-Champaign
Abstract
This is the second year of the TREC Deep Learning Track, with the goal of studying ad hoc ranking in the large training data regime. We again have a document retrieval task and a passage retrieval task, each with hundreds of thousands of human-labeled training queries. We evaluate using single-shot TREC-style evaluation, to give us a picture of which ranking methods work best when large data is available, with much more comprehensive relevance labeling on the small number of test queries. This year we have further evidence that rankers with BERT-style pretraining outperform other rankers in the large data regime.
中文速览
在信息检索领域，当训练数据规模足够大时，哪类排序方法表现最好，这一问题此前缺乏系统性验证。TREC 2020深度学习赛道（Deep Learning Track）围绕MS MARCO构建了包含数十万人工标注查询的大规模训练集，并分别设置了文档检索和段落检索两项任务，由NIST提供比MS MARCO稀疏标注更全面的测试集评估标准。结果显示，基于BERT式预训练语言模型（pretrained neural language model，nnlm）的排序方法在两项任务中均大幅领先传统检索方法（如BM25），在文档检索和段落检索中最优nnlm模型分别比最优传统方法高出23%和42%的NDCG@10；同时，端到端全量检索（full ranking）相比在给定候选集上重排序（reranking）并未呈现出明显优势。这项工作不仅为大规模训练数据条件下深度学习排序方法的有效性提供了迄今最有力的实证证据，其构建的可复用测试集也将为后续研究提供标准化的评测基准。
原文 arXiv:2102.07662;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.07662v1