Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?
Nenad Tomasev§ DeepMind London, UK Ioana Bica§ DeepMind London, UK Brian McWilliams§, * Twitter Cortex Lars Buesing DeepMind London, UK Razvan Pascanu DeepMind London, UK Charles Blundell DeepMind London, UK Jovana Mitrovic§,††\dagger DeepMind London, UK
Abstract
Despite recent progress made by self-supervised methods in representation learning with residual networks, they still underperform supervised learning on the ImageNet classification benchmark, limiting their applicability in performance-critical settings. Building on prior theoretical insights from ReLIC [Mitrovic et al., 2021], we include additional inductive biases into self-supervised learning. We propose a new self-supervised representation learning method, ReLICv2, which combines an explicit invariance loss with a contrastive objective over a varied set of appropriately constructed data views to avoid learning spurious correlations and obtain more informative representations. ReLICv2 achieves $77.1\%$ top- $1$ accuracy on ImageNet under linear evaluation on a ResNet50, thus improving the previous state-of-the-art by absolute $+1.5\%$ ; on larger ResNet models, ReLICv2 achieves up to $80.6\%$ outperforming previous self-supervised approaches with margins up to $+2.3\%$ . Most notably, ReLICv2 is the first unsupervised representation learning method to consistently outperform the supervised baseline in a like-for-like comparison over a range of ResNet architectures. Using ReLICv
中文速览
自监督表示学习长期无法超越有监督方法的核心症结，在于模型会把背景、风格等与下游任务无关的"虚假特征"也编入表征。研究者在ReLIC方法的基础上提出ReLICv2，引入两项额外归纳偏置：一是全无监督的显著性掩码流水线，能在不借助任何标注的情况下把前景与背景剥离，驱使模型专注于语义相关区域；二是多尺度多视图策略，同时使用大量全图视图（增强风格不变性）和少量小裁剪视图（模拟遮挡、迫使模型从局部推断整体），并将对比损失与不变性损失统一扩展到这一多视图框架上。实验结果显示，ReLICv2在ResNet50上的ImageNet线性评估准确率达到77.1%，比此前最优自监督方法高出1.5个百分点，在更大的ResNet200 2×上更达到80.6%；更关键的是，它成为首个在一系列不同深度和宽度的ResNet架构上全面超越同等规模有监督基线的自监督方法，同时在迁移学习、分布外泛化和语义分割等任务上也全面领先，意义在于证明了无标注预训练在性能敏感场景中的实际可行性。
原文 arXiv:2201.05119;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2201.05119v2