Distill-VQ: Learning Retrieval Oriented Vector Quantization By Distilling Knowledge from Dense EmbeddingsConference: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 11–15, 2022; Madrid, SpainProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ’22), July 11–15, 2022, Madrid, SpainPrice: 15.00DOI: 10.1145/3477495.3531799ISBN: 978-1-4503-8732-3/22/07CCS: Computing methodologies Continuous space search
Shitao Xiao♠{}^{\textbf{{\tiny♠}}}, Zheng Liu♣{}^{\textbf{{\tiny♣}}}, Weihao Han♠{}^{\textbf{{\tiny♠}}}, Jianjin Zhang♠{}^{\textbf{{\tiny♠}}}, Defu Lian♥{}^{\textbf{{\tiny♥}}}, Yeyun Gong♣{}^{\textbf{{\tiny♣}}}, Qi Chen♣{}^{\textbf{{\tiny♣}}}, Fan Yang♣{}^{\textbf{{\tiny♣}}}, Hao Sun♠{}^{\textbf{{\tiny♠}}}, Yingxia Shao♠{}^{\textbf{{\tiny♠}}}, Xing Xie♣{}^{\textbf{{\tiny♣}}} Affiliation: ♠: Beijing University of Posts and Telecommunications , China Affiliation: ♣: Microsoft Research Asia , China Affiliation: ♠: Microsoft Search Technology Center Asia , China Affiliation: ♥: University of Science and Technology of China , China email: email: email:
Abstract
Vector quantization (VQ) based ANN indexes, such as Inverted File System (IVF) and Product Quantization (PQ), have been widely applied to embedding based document retrieval thanks to the competitive time and memory efficiency. Originally, VQ is learned to minimize the reconstruction loss, i.e., the distortions between the original dense embeddings and the reconstructed embeddings after quantization. Unfortunately, such an objective is inconsistent with the goal of selecting ground-truth documents for the input query, which may cause severe loss of retrieval quality. Recent works identify such a defect, and propose to minimize the retrieval loss through contrastive learning. However, these methods intensively rely on queries with ground-truth documents, whose performance is limited by the insufficiency of labeled data.
原文 arXiv:2204.00185;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.00185v2