Poly-encoders: architectures and pre-training strategies for fast and accurate multi-sentence scoring
Samuel Humeau , Kurt Shuster∗, Marie-Anne Lachaux, Jason Weston Facebook AI Research Joint First Authors.
Abstract
The use of deep pre-trained transformers has led to remarkable progress in a number of applications (Devlin et al., 2019). For tasks that make pairwise comparisons between sequences, matching a given input with a corresponding label, two approaches are common: Cross-encoders performing full self-attention over the pair and Bi-encoders encoding the pair separately. The former often performs better, but is too slow for practical use. In this work, we develop a new transformer architecture, the Poly-encoder, that learns global rather than token level self-attention features. We perform a detailed comparison of all three approaches, including what pre-training and fine-tuning strategies work best. We show our models achieve state-of-the-art results on four tasks; that Poly-encoders are faster than Cross-encoders and more accurate than Bi-encoders; and that the best results are obtained by pre-training on large datasets similar to the downstream tasks.
中文速览
如何在保持准确率的同时让候选答案检索足够快，是对话和信息检索系统的核心痛点：Cross-encoder（交叉编码器）精度高但每次都要把输入和候选拼在一起重新算，速度慢得无法落地；Bi-encoder（双编码器）可以预先缓存候选向量、速度极快，但准确率明显偏低。研究者提出了一种新架构 Poly-encoder（多码编码器），它在双编码器的基础上引入一组可学习的全局注意力码本，让输入端产生多个全局特征向量，再与候选向量做轻量级的交叉注意力融合，从而在不需要每次重算候选表示的前提下捕捉到更丰富的上下文-候选交互信息。实验在对话（ConvAI2、DSTC7、Ubuntu V2）和信息检索（Wikipedia文章搜索）四项任务上均刷新了最优成绩，Poly-encoder 的准确率显著超过 Bi-encoder，推理速度又远快于 Cross-encoder，同时研究还发现用与下游任务领域更接近的 Reddit 数据预训练比通用 BERT 预训练带来更大的提升。这项工作为需要在海量候选中实时检索的实际系统提供了一个兼顾质量与速度的实用解决方案。
原文 arXiv:1905.01969;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1905.01969v4