Learning Dense Representations of Phrases at Scale
Jinhyuk Lee1,2 Mujeen Sung1 Jaewoo Kang1 Danqi Chen2 Korea University1 Princeton University2 Work partly done while visiting Princeton University.
Abstract
Open-domain question answering can be reformulated as a phrase retrieval problem, without the need for processing documents on-demand during inference (Seo et al., 2019). However, current phrase retrieval models heavily depend on sparse representations and still underperform retriever-reader approaches. In this work, we show for the first time that we can learn dense representations of phrases alone that achieve much stronger performance in open-domain QA. We present an effective method to learn phrase representations from the supervision of reading comprehension tasks, coupled with novel negative sampling methods. We also propose a query-side fine-tuning strategy, which can support transfer learning and reduce the discrepancy between training and inference. On five popular open-domain QA datasets, our model DensePhrases improves over previous phrase retrieval models by $15\%$ – $25\%$ absolute accuracy and matches the performance of state-of-the-art retriever-reader models. Our model is easy to parallelize due to pure dense representations and processes more than 10 questions per second on CPUs. Finally, we directly use our pre-indexed dense phrase representations for two slot fil
中文速览
开放域问答长期依赖"先检索文档、再用阅读理解模型精读"的两阶段流程，计算开销大且容易因检索失误而连锁出错。DensePhrases 把这个问题转化为"直接在海量短语向量里做相似度搜索"——把维基百科里的所有短语（约600亿个）预先编码成稠密向量存起来，回答问题时只需把问题也编码成向量，一次向量检索就能拿到答案短语，完全不需要在线处理文档。为了让纯稠密表示达到足够高的质量，研究者结合了数据增强、知识蒸馏、批内负采样以及一种新提出的"前序批次负采样"来训练短语编码器，并引入只微调问题端编码器的迁移学习策略来弥合训练与推理之间的差距。最终在五个主流开放域问答数据集上，DensePhrases 比此前基于短语检索的方法提升了15%–25%的绝对准确率，与最强的两阶段检索-阅读模型持平，同时在CPU上每秒可处理逾10个问题，存储空间也从1.5TB压缩到320GB，还能零改动地迁移到槽位填充等知识密集型任务，为构建高效的稠密知识库提供了切实可行的新路径。
原文 arXiv:2012.12624;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.12624v3