BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA
Nora Kassner, Hinrich Schütze Center for Information and Language Processing (CIS) LMU Munich, Germany
Abstract
Khandelwal et al. (2020) use a k-nearest-neighbor (kNN) component to improve language model performance. We show that this idea is beneficial for open-domain question answering (QA). To improve the recall of facts encountered during training, we combine BERT Devlin et al. (2019) with a traditional information retrieval step (IR) and a kNN search over a large datastore of an embedded text collection. Our contributions are as follows: i) BERT-kNN outperforms BERT on cloze-style QA by large margins without any further training. ii) We show that BERT often identifies the correct response category (e.g., US city), but only kNN recovers the factually correct answer (e.g., “Miami”). iii) Compared to BERT, BERT-kNN excels for rare facts. iv) BERT-kNN can easily handle facts not covered by BERT’s training set, e.g., recent events.
中文速览
预训练语言模型（如BERT）虽然读过海量文本，却在开放域问答上表现欠佳，原因在于它难以精确"回忆"训练中见过的具体事实。为此，研究者提出了BERT-kNN：先用传统信息检索（IR）锁定与问题相关的维基百科段落，再在这些段落的BERT嵌入向量上做k近邻（kNN）搜索，最后将检索结果与BERT原始预测按比例融合，整个过程无需任何额外训练。实验表明，BERT-kNN在LAMA完形填空问答基准上比BERT-base和BERT-large均高出约10个百分点，在剔除易猜事实的LAMA-UHN子集上优势更为显著，对罕见事实和BERT训练期后的新事件（如2020年新增的维基词条）同样有效。这项工作说明，把检索记忆机制插入预训练模型是一种轻量、灵活的知识增强路径，不仅能弥补模型"知道类别却猜错具体答案"的短板，还能让模型随时纳入新知识而无需重新训练。
原文 arXiv:2005.00766;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.00766v2