kNN-Prompt: Nearest Neighbor Zero-Shot Inference
Weijia Shi Julian Michael Suchin Gururangan Luke Zettlemoyer Paul G. Allen School of Computer Science、Engineering, University of Washington, Seattle, WA {swj0419, julianjm, sg01,
Abstract
Retrieval-augmented language models (LMs) use non-parametric memory to substantially outperform their non-retrieval counterparts on perplexity-based evaluations, but it is an open question whether they achieve similar gains in few- and zero-shot end-task accuracy. We extensively study one such model, the $k$ -nearest neighbor LM ( $k$ NN-LM), showing that the gains marginally transfer. The main challenge is to achieve coverage of the verbalizer tokens that define the different end-task class labels. To address this challenge, we also introduce kNN-Prompt, a simple and effective $k$ NN-LM with automatically expanded fuzzy verbalizers (e.g. to expand “terrible” to also include “silly” and other task-specific synonyms for sentiment classification). Across nine diverse end-tasks, using kNN-Prompt with GPT-2 large yields significant performance boosts over strong zero-shot baselines (13.4% absolute improvement over the base LM on average). We also show that other advantages of non-parametric augmentation hold for end tasks; kNN-Prompt is effective for domain adaptation with no further training, and gains increase with the size of the retrieval model.
中文速览
检索增强语言模型（retrieval-augmented LM）在语言建模困惑度指标上表现优异，但能否在零样本（zero-shot）分类任务中带来同等收益，此前一直不清楚。本文以 kNN-LM 为基础，发现其直接用于零样本分类时提升甚微，核心瓶颈在于 kNN 分布极度稀疏，往往无法覆盖用于判断类别的关键词（verbalizer token）。为此，研究者提出了 kNN-Prompt，通过结合 GloVe 词向量相似度和 ConceptNet 同义词，自动为每个类别标签构建"模糊标签词集合"（fuzzy verbalizer），并配合领域条件 PMI 校准，让模型得以充分利用检索到的近邻信息。在九项涵盖情感分析、主题分类、文本蕴含等多样化任务上，kNN-Prompt 基于 GPT-2 Large 平均绝对提升达 13.4%，还无需额外训练即可实现领域自适应，且模型越大收益越显著。这项工作揭示了检索增强与零样本推理结合的可行路径，为将来在更大规模语言模型上进一步挖掘检索潜力提供了有力依据。
原文 arXiv:2205.13792;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.13792v2