SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval
Yang Bai Tsinghua University , Xiaoguang Li Huawei Noah’s Ark Lab , Gang Wang Huawei Noah’s Ark Lab , Chaoliang Zhang Huawei Noah’s Ark Lab , Lifeng Shang Huawei Noah’s Ark Lab , Jun Xu Renmin University of China , Zhaowei Wang Huawei Noah’s Ark Lab , Fangshan Wang Huawei Technologies Co., Ltd and Qun Liu Huawei Noah’s Ark Lab
Abstract
Term-based sparse representations dominate the first-stage text retrieval in industrial applications, due to its advantage in efficiency, interpretability, and exact term matching. In this paper, we study the problem of transferring the deep knowledge of the pre-trained language model (PLM) to Term-based Sparse representations, aiming to improve the representation capacity of bag-of-words(BoW) method for semantic-level matching, while still keeping its advantages. Specifically, we propose a novel framework SparTerm to directly learn sparse text representations in the full vocabulary space. The proposed SparTerm comprises an importance predictor to predict the importance for each term in the vocabulary, and a gating controller to control the term activation. These two modules cooperatively ensure the sparsity and flexibility of the final text representation, which unifies the term-weighting and expansion in the same framework. Evaluated on MSMARCO dataset, SparTerm significantly outperforms traditional sparse methods and achieves state of the art ranking performance among all the PLM-based sparse models.
中文速览
传统词袋（Bag-of-Words, BoW）检索方法虽然高效、可解释、支持精确匹配，却在语义层面的匹配上存在明显短板，而现有的改进方案（如DeepCT、Doc2Query）只是用预训练语言模型（PLM）辅助调整稀疏表示，并未从根本上打通两者。本文提出SparTerm框架，直接在整个词汇表空间中学习基于词项的稀疏表示：用一个"重要性预测器"借助BERT的上下文能力为词汇表中每个词赋予语义重要性权重，再用一个"门控控制器"以二值信号决定哪些词（包括原文中没有出现的扩展词）最终参与表示，从而将词权重调整与词项扩展统一在同一框架内。在MS MARCO数据集上的实验表明，SparTerm显著优于以往所有基于PLM的稀疏检索模型，甚至超过了模型规模大2倍、预训练语料大70倍的Doc2Query-T5，证明稀疏检索方法还有巨大的性能提升空间，同时保留了BoW固有的效率与可解释性优势。
原文 arXiv:2010.00768;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.00768v1