Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense Embeddings
Shitao Xiao♠♠{}^{\textbf{{\tiny\char 171}}}, Zheng Liu♣♣{}^{\textbf{{\tiny\char 168}}}, Weihao Han♠♠{}^{\textbf{{\tiny\char 169}}}, Jianjin Zhang♠♠{}^{\textbf{{\tiny\char 169}}}, Defu Lian♥♥{}^{\textbf{{\tiny\char 170}}}, Yeyun Gong♣♣{}^{\textbf{{\tiny\char 168}}}, Qi Chen♣♣{}^{\textbf{{\tiny\char 168}}}, Fan Yang♣♣{}^{\textbf{{\tiny\char 168}}}, Hao Sun♠♠{}^{\textbf{{\tiny\char 169}}}, Yingxia Shao♠♠{}^{\textbf{{\tiny\char 171}}}, Xing Xie♣♣{}^{\textbf{{\tiny\char 168}}} ♠: Beijing University of Posts and Telecommunications China ♣: Microsoft Research Asia China ♠: Microsoft Search Technology Center Asia China ♥: University of Science and Technology of China China
Abstract
Vector quantization (VQ) based ANN indexes, such as Inverted File System (IVF) and Product Quantization (PQ), have been widely applied to embedding based document retrieval thanks to the competitive time and memory efficiency. Originally, VQ is learned to minimize the reconstruction loss, i.e., the distortions between the original dense embeddings and the reconstructed embeddings after quantization. Unfortunately, such an objective is inconsistent with the goal of selecting ground-truth documents for the input query, which may cause severe loss of retrieval quality. Recent works identify such a defect, and propose to minimize the retrieval loss through contrastive learning. However, these methods intensively rely on queries with ground-truth documents, whose performance is limited by the insufficiency of labeled data.
中文速览
向量量化（Vector Quantization, VQ）是加速大规模文档检索的核心技术，但传统方法用"重建误差最小化"来训练量化模型，与"找到相关文档"这一真实目标并不一致，而近年改用对比学习来直接优化检索质量的方法又严重依赖大量人工标注数据。本文提出 Distill-VQ，将 IVF 和 PQ 两个量化模块的联合学习纳入知识蒸馏（knowledge distillation）框架：以训练好的稠密向量编码器作为"教师"，让它对每个查询预测各候选文档的相关分数，再以量化模块作为"学生"去模仿这些分数，从而迫使压缩后的向量尽可能保留原始编码器的检索能力。这一设计的关键优势在于无需人工标注——无标签数据同样能提供丰富的训练信号，教师给出的高相关性文档自动充当正样本，低相关文档则被施以梯度惩罚，有效缓解了假负例问题。在 MS MARCO 和 Natural Questions 两个大型基准上，Distill-VQ 在 Recall 和 MRR 指标上均明显超越现有最优量化方法，且可直接插入 FAISS 等工业级检索框架，对需要在标注资源有限场景下部署高效向量检索的实际应用具有重要意义。
原文 arXiv:2204.00185;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.00185v2