ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer
Yuanmeng Yan1 , Rumei Li2∗, Sirui Wang2, Fuzheng Zhang2, Wei Wu2, Weiran Xu1 1Beijing University of Posts and Telecommunications, Beijing, China 2Meituan Inc., Beijing, China Work done during internship at Meituan Inc. The first two authors contribute equally. Weiran Xu is the corresponding author.
Abstract
Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised SEntence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.
中文速览
BERT虽然在众多NLP任务上表现出色，但它直接输出的句子向量存在严重的"坍缩"问题——几乎所有句子都被映射到一个狭小的空间里，导致语义相似度计算效果极差。为此，作者提出了ConSERT，一个基于对比学习（contrastive learning）的无监督句子表示迁移框架：将同一句子通过对抗扰动、词序打乱、随机遮蔽、Dropout等方式生成两个不同的"视角"，训练BERT让同句的两个视角在向量空间中相互靠近、与其他句子远离，从而重塑句子表示的分布。实验表明，仅用无标注目标域文本微调，ConSERT在语义文本相似度（STS）基准上比此前最优方法提升约8%，甚至可媲美使用了标注数据的SBERT-NLI；若进一步引入NLI标注数据，则刷新了STS任务的最优成绩，且在仅有1000条样本时仍保持强劲表现。这项工作的意义在于，它提供了一种轻量、高效、无需额外标注的方案来修复BERT句子表示的根本缺陷，极大降低了在实际场景中部署高质量语义匹配系统的门槛。
原文 arXiv:2105.11741;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.11741v1