InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers
Leonid Boytsov*{}^{*}start_FLOATSUPERSCRIPT * end_FLOATSUPERSCRIPT Amazon AWS AI Labs Pittsburgh USA Preksha Patel Vivek Sourabh Riddhi Nisar Sayani Kundu Ramya Ramanathan Eric Nyberg Carnegie Mellon University Pittsburgh USA
Abstract
We carried out a reproducibility study of InPars, which is a method for unsupervised training of neural rankers (Bonifacio et al., 2022). As a by-product, we developed InPars-light, which is a simple-yet-effective modification of InPars. Unlike InPars, InPars-light uses 7x-100x smaller ranking models and only a freely available language model BLOOM, which—as we found out—produced more accurate rankers compared to a proprietary GPT-3 model. On all five English retrieval collections (used in the original InPars study) we obtained substantial (7%-30%) and statistically significant improvements over BM25 (in nDCG and MRR) using only a 30M parameter six-layer MiniLM-30M ranker and a single three-shot prompt. In contrast, in the InPars study only a 100x larger monoT5-3B model consistently outperformed BM25, whereas their smaller monoT5-220M model (which is still 7x larger than our MiniLM ranker) outperformed BM25 only on MS MARCO and TREC DL 2020. In the same three-shot prompting scenario, our 435M parameter DeBERTA v3 ranker was at par with the 7x larger monoT5-3B (average gain over BM25 of 1.3 vs 1.32): In fact, on three out of five datasets, DeBERTA slightly outperformed monoT5-3B. Fi
中文速览
训练神经网络排序模型通常需要大量人工标注数据，成本极高，而现有基于大语言模型生成合成训练数据的方案（如InPars）要么依赖昂贵的专有模型，要么需要数十亿参数的超大排序模型，同样不够经济实用。本文对InPars方法进行了严格的可复现性研究，并在此过程中提出了改进版本InPars-light：用完全开源、免费的BLOOM语言模型替代收费的GPT-3来生成合成查询，同时引入一致性检验过滤噪声样本，并采用仅3000万参数的轻量级MiniLM模型作为重排序器。实验结果表明，InPars-light在五个英文信息检索数据集上均取得了相对BM25基线7%至30%的显著提升，而原始InPars需要比我们大100倍的monoT5-3B模型才能达到类似的全面超越效果。这项工作的意义在于，它首次证明无需昂贵的专有模型和超大排序器，仅凭开源工具和小模型就能构建出真正低成本、可实际部署的无监督神经排序系统。
原文 arXiv:2301.02998;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2301.02998v2