Zero-Shot Listwise Document Reranking with a Large Language Model
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, Jimmy Lin David R. Cheriton School of Computer Science, University of Waterloo, Canada {x93ma, x978zhang, rpradeep,
Abstract
Supervised ranking methods based on bi-encoder or cross-encoder architectures have shown success in multi-stage text ranking tasks, but they require large amounts of relevance judgments as training data. In this work, we propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data. Different from the existing pointwise ranking methods, where documents are scored independently and ranked according to the scores, LRL directly generates a reordered list of document identifiers given the candidate documents. Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker to improve the top-ranked results of a pointwise method for improved efficiency. Additionally, we apply our approach to subsets of MIRACL, a recent multilingual retrieval dataset, with results showing its potential to generalize across different languages.111This work was conducted in late 2022 and was rejected from ACL 2023.
中文速览
大规模语言模型在信息检索中的重排序（reranking）通常依赖海量人工标注数据，这在低资源或跨语言场景下代价高昂。研究者提出了LRL（Listwise Reranker with a Large Language Model），直接用GPT-3将查询和一批候选文档一起输入，让模型按相关度生成一个文档编号的排列列表，而不是像传统方法那样给每篇文档单独打分再排序。在三个TREC网络搜索数据集上，LRL比当时最优的零样本逐点打分方法UPR平均高出约6个nDCG@10分，还能叠加在已有排序结果之上作为最终精排阶段进一步提升效果；在中文、斯瓦希里语、约鲁巴语三个多语言数据集上同样超越了基线，显示出跨语言泛化潜力。这项工作表明，无需任何任务专属训练数据，仅凭大语言模型的列表式推理能力就能实现接近乃至超越部分有监督模型的检索精度，为零样本信息检索提供了一条实用路径。
原文 arXiv:2305.02156;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.02156v1