Fine-Tuning LLaMA for Multi-Stage Text Retrieval
Xueguang Ma ††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT Liang Wang ‡‡{}^{\ddagger}start_FLOATSUPERSCRIPT ‡ end_FLOATSUPERSCRIPT Nan Yang ‡‡{}^{\ddagger}start_FLOATSUPERSCRIPT ‡ end_FLOATSUPERSCRIPT Furu Wei ‡‡{}^{\ddagger}start_FLOATSUPERSCRIPT ‡ end_FLOATSUPERSCRIPT Jimmy Lin ††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT ††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT David R. Cheriton School of Computer Science, University of Waterloo ‡‡{}^{\ddagger}start_FLOATSUPERSCRIPT ‡ end_FLOATSUPERSCRIPT Microsoft Research
Abstract
The effectiveness of multi-stage text retrieval has been solidly demonstrated since before the era of pre-trained language models. However, most existing studies utilize models that predate recent advances in large language models (LLMs). This study seeks to explore potential improvements that state-of-the-art LLMs can bring. We conduct a comprehensive study, fine-tuning the latest LLaMA model both as a dense retriever (RepLLaMA) and as a pointwise reranker (RankLLaMA) for both passage retrieval and document retrieval using the MS MARCO datasets. Our findings demonstrate that the effectiveness of large language models indeed surpasses that of smaller models. Additionally, since LLMs can inherently handle longer contexts, they can represent entire documents holistically, obviating the need for traditional segmenting and pooling strategies. Furthermore, evaluations on BEIR demonstrate that our RepLLaMA–RankLLaMA pipeline exhibits strong zero-shot effectiveness. Model checkpoints from this study are available on HuggingFace.111https://huggingface.co/castorini
中文速览
多阶段文本检索（retrieval）一直是搜索和问答系统的核心技术，但此前的检索器和重排器大多基于BERT这类较小的预训练模型，尚未充分利用近年涌现的大语言模型（LLM）。研究者将开源的LLaMA-2模型分别微调为双编码密集检索器（RepLLaMA）和逐点相关性重排器（RankLLaMA），构建起一套完整的两阶段检索流水线，并在MS MARCO段落与文档检索任务及跨领域零样本基准BEIR上进行了系统评测。实验结果表明，RepLLaMA–RankLLaMA流水线在检索和重排两个环节均超越了此前所有已知系统，且因LLM天然支持较长上下文，可以直接对完整文档建模，无需传统的分段再聚合策略。这项工作证明了将大语言模型直接微调用于检索任务的可行性与优越性，为后续在检索增强生成（RAG）等场景中部署更强大的检索系统提供了重要参考。
原文 arXiv:2310.08319;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.08319v1