Nonparametric Masked Language Modeling
Sewon Min1,2 Weijia Shi1,2 Mike Lewis2 Xilun Chen2 Wen-tau Yih2 Hannaneh Hajishirzi1,3 Luke Zettlemoyer1,2 1University of Washington 2Meta AI 3Allen Institute for AI
Abstract
Existing language models (LMs) predict tokens with a softmax over a finite vocabulary, which can make it difficult to predict rare tokens or phrases. We introduce NpM, the first nonparametric masked language model that replaces this softmax with a nonparametric distribution over every phrase in a reference corpus. NpM fills in the [MASK] solely from retrieving a token from a text corpus. We show that NpM can be efficiently trained with a contrastive objective and an in-batch approximation to full corpus retrieval. Zero-shot evaluation on 16 tasks including classification, fact probing and question answering demonstrates that NpM outperforms significantly larger parametric models, either with or without a retrieve-and-generate approach. It is particularly better at dealing with rare patterns (word senses or facts) and predicting rare or nearly unseen words (e.g., non-Latin script). We release the model and code at github.com/facebookresearch/NPM.
中文速览
用软件词表来预测词语时，稀有词和未登录词往往很难被正确输出——比如多子词拼合的地名或非拉丁文字。NpM（非参数掩码语言模型）从根本上抛弃了传统的 softmax 固定词表，改为让模型直接从一个参考语料库里检索任意短语来填充被遮住的位置，输出空间因此不再受词表大小约束。训练时，NpM 用批内近似替代全语料检索，并以短语级对比学习目标来驱动编码器学会把掩码位置的向量对齐到正确短语的起止位置。在 16 项零样本任务（分类、事实探测、问答）上的测评显示，NpM 显著优于参数量多出数百倍的传统参数模型及检索增强生成模型，尤其在处理稀有词义、罕见实体名称和非拉丁文字预测上表现突出。这项工作证明，彻底去掉输出端 softmax、转而依赖可随时替换和扩展的语料库，是提升语言模型长尾知识表达能力的一条切实可行的路径。
原文 arXiv:2212.01349;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.01349v2