Fast, Effective, and Self-Supervised: Transforming Masked Language Models into Universal Lexical and Sentence Encoders
Fangyu Liu, Ivan Vulić, Anna Korhonen, Nigel Collier Language Technology Lab, TAL, University of Cambridge {fl399, iv250, alk23, nhc30}cam.ac.uk
Abstract
Previous work has indicated that pretrained Masked Language Models (MLMs) are not effective as universal lexical and sentence encoders off-the-shelf, i.e., without further task-specific fine-tuning on NLI, sentence similarity, or paraphrasing tasks using annotated task data. In this work, we demonstrate that it is possible to turn MLMs into effective lexical and sentence encoders even without any additional data, relying simply on self-supervision. We propose an extremely simple, fast, and effective contrastive learning technique, termed Mirror-BERT, which converts MLMs (e.g., BERT and RoBERTa) into such encoders in 20–30 seconds with no access to additional external knowledge. Mirror-BERT relies on identical and slightly modified string pairs as positive (i.e., synonymous) fine-tuning examples, and aims to maximise their similarity during “identity fine-tuning”. We report huge gains over off-the-shelf MLMs with Mirror-BERT both in lexical-level and in sentence-level tasks, across different domains and different languages. Notably, in sentence similarity (STS) and question-answer entailment (QNLI) tasks, our self-supervised Mirror-BERT model even matches the performance of the Sent
中文速览
预训练语言模型（如BERT）虽然存储了大量语义知识，却无法直接用作高质量的词汇或句子编码器，通常需要在人工标注数据上进一步微调才能胜任相似度等任务。为此，研究者提出了Mirror-BERT：一种完全自监督的对比学习方法，无需任何外部标注数据，仅靠将输入文本自我复制并配合随机遮盖和dropout两种简单数据增强手段生成正样本对，就能在20到30秒内把BERT/RoBERTa改造成通用语言编码器。实验表明，Mirror-BERT在词汇相似度、生物医学实体链接、多语言句子相似度（STS）及问答蕴含（QNLI）等任务上大幅超越原始MLM，在STS和QNLI上甚至能媲美依赖人工标注数据训练的Sentence-BERT。这一结果说明MLM内部已潜藏丰富的语义结构，只需极低成本的自监督"触发"便可将其释放，对无标注资源场景下的NLP应用具有重要实践价值。
原文 arXiv:2104.08027;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.08027v2