Matching-oriented Product Quantization For Ad-hoc Retrieval
Shitao Xiao1, Zheng Liu2, Yingxia Shao, Defu Lian3, Xing Xie2 1: Beijing University of Posts and Telecommunications, Beijing, China 2: Microsoft Research Asia, Beijing, China 3: University of Science and Technology of China, Hefei, China Work is done during the internship at Microsoft.Corresponding author.
Abstract
Product quantization (PQ) is a widely used technique for ad-hoc retrieval. Recent studies propose supervised PQ, where the embedding and quantization models can be jointly trained with supervised learning. However, there is a lack of appropriate formulation of the joint training objective; thus, the improvements over previous non-supervised baselines are limited in reality. In this work, we propose the Matching-oriented Product Quantization (MoPQ), where a novel objective Multinoulli Contrastive Loss (MCL) is formulated. With the minimization of MCL, we are able to maximize the matching probability of query and ground-truth key, which contributes to the optimal retrieval accuracy. Given that the exact computation of MCL is intractable due to the demand of vast contrastive samples, we further propose the Differentiable Cross-device Sampling (DCS), which significantly augments the contrastive samples for precise approximation of MCL. We conduct extensive experimental studies on four real-world datasets, whose results verify the effectiveness of MoPQ. The code is available at https://github.com/microsoft/MoPQ.
中文速览
产品量化（Product Quantization, PQ）是信息检索中压缩向量、加速搜索的主流技术，但现有有监督PQ方法普遍沿用"重建误差最小化"作为训练目标，这篇论文从理论和实验两个角度证明这个目标存在根本性缺陷——量化误差不可能被彻底消除，而且降低重建误差并不一定提升检索精度。为此，作者提出了"面向匹配的产品量化"（MoPQ），核心是一个新损失函数——多努利对比损失（Multinoulli Contrastive Loss, MCL），它将检索过程建模为概率生成过程，直接最大化query与正确候选item的匹配概率，从而在理论上保证最优检索精度。由于MCL的精确计算需要海量负样本，作者进一步提出了跨设备可微采样（Differentiable Cross-device Sampling, DCS）机制，通过分布式共享嵌入显著扩充对比样本，并用"主损失与镜像损失组合"技巧解决梯度截断导致的更新失真问题。在四个真实文本检索数据集上，MoPQ显著超越现有有监督PQ基线，为如何正确制定PQ联合训练目标提供了清晰的理论依据和实用方案。
原文 arXiv:2104.07858;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.07858v3