Text Transformations in Contrastive Self-Supervised Learning: A Review
Amrita Bhattacharjee111Authors contributed equally to this work. Names are in alphabetical order. Mansooreh Karami∗、Huan Liu Arizona State University, Tempe, AZ {abhatt43, mkarami,
Abstract
Contrastive self-supervised learning has become a prominent technique in representation learning. The main step in these methods is to contrast semantically similar and dissimilar pairs of samples. However, in the domain of Natural Language Processing (NLP), the augmentation methods used in creating similar pairs with regard to contrastive learning (CL) assumptions are challenging. This is because, even simply modifying a word in the input might change the semantic meaning of the sentence, and hence, would violate the distributional hypothesis. In this review paper, we formalize the contrastive learning framework, emphasize the considerations that need to be addressed in the data transformation step, and review the state-of-the-art methods and evaluations for contrastive representation learning in NLP. Finally, we describe some challenges and potential directions for learning better text representations using contrastive methods.
中文速览
对比自监督学习（contrastive self-supervised learning）在图像领域已取得显著成效，但将其迁移到自然语言处理（NLP）时面临一个核心难题：文本的离散性使得"数据增强"极为棘手——哪怕只改动一个词，句子的语义就可能发生根本变化，从而破坏对比学习所依赖的"正样本语义相似"这一前提。这篇综述系统梳理了对比表示学习在NLP中的研究现状，从正式框架出发，重点整理了输入空间变换、潜在空间变换、对抗扰动等多类数据增强策略，以及困难负样本（hard negative）挖掘等关键技术，同时总结了常用损失函数和评估指标。研究发现，现有方法在正负样本的构造上仍存在语义一致性难以保证、采样偏差等系统性挑战，并指出了未来改进文本表示的若干有价值方向。这项工作为研究者快速把握NLP对比学习全貌、识别开放问题提供了一份系统性的路线图。
原文 arXiv:2203.12000;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.12000v2