Supervised Contrastive Learning
Prannay Khosla Google Research Equal contribution. Piotr Teterwak Boston University Work done while at Google Research. Chen Wang Snap Inc. Aaron Sarna Google Research Corresponding author: Yonglong Tian MIT Phillip Isola MIT Aaron Maschinot Google Research Ce Liu Google Research Dilip Krishnan Google Research
Abstract
Contrastive learning applied to self-supervised representation learning has seen a resurgence in recent years, leading to state of the art performance in the unsupervised training of deep image models. Modern batch contrastive approaches subsume or significantly outperform traditional contrastive losses such as triplet, max-margin and the N-pairs loss. In this work, we extend the self-supervised batch contrastive approach to the fully-supervised setting, allowing us to effectively leverage label information. Clusters of points belonging to the same class are pulled together in embedding space, while simultaneously pushing apart clusters of samples from different classes. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of $81.4\%$ on the ImageNet dataset, which is $0.8\%$ above the best number reported for this architecture. We show consistent outperformance over cross-entropy on other datasets and two ResNet variants. The loss shows benefits for robustness to natural corruptions, and is more stable to hyperparameter settings such as optimizers and data aug
中文速览
监督对比学习（Supervised Contrastive Learning，SupCon）要解决的问题是：主流的交叉熵损失函数在大规模图像分类中虽然好用，但对噪声标签鲁棒性差、泛化边界也不够理想。研究者将近年在自监督领域大放异彩的对比学习扩展到有标签的全监督场景，核心思路是：对每个样本（锚点），不只用一个正样本，而是把同类的所有样本都拉近，同时把不同类的样本推远，从而提出了SupCon损失函数。在ImageNet数据集上，使用ResNet-200时达到了81.4%的Top-1准确率，比同架构下最优的交叉熵方法高出0.8%，并在多个数据集和网络结构上稳定超越交叉熵基线。这项工作的重要性在于：SupCon不仅提升了分类精度，还增强了模型对自然噪声的鲁棒性，对超参数设置更不敏感，且实现简单，为有监督和自监督学习提供了一个统一的损失函数框架。
原文 arXiv:2004.11362;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.11362v5