Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
Mathilde Caron1,2、Ishan Misra2、Julien Mairal1 Priya Goyal2、Piotr Bojanowski2、Armand Joulin2 1 Inria∗、2 Facebook AI Research
Abstract
Unsupervised image representations have significantly reduced the gap with supervised pretraining, notably with the recent achievements of contrastive learning methods. These contrastive methods typically work online and rely on a large number of explicit pairwise feature comparisons, which is computationally challenging. In this paper, we propose an online algorithm, SwAV, that takes advantage of contrastive methods without requiring to compute pairwise comparisons. Specifically, our method simultaneously clusters the data while enforcing consistency between cluster assignments produced for different augmentations (or “views”) of the same image, instead of comparing features directly as in contrastive learning. Simply put, we use a “swapped” prediction mechanism where we predict the code of a view from the representation of another view. Our method can be trained with large and small batches and can scale to unlimited amounts of data. Compared to previous contrastive methods, our method is more memory efficient since it does not require a large memory bank or a special momentum network. In addition, we also propose a new data augmentation strategy, multi-crop, that uses a mix of v
中文速览
自监督视觉表征学习一直依赖对比学习，但对比学习需要对海量图像对逐一比较特征，计算代价极高。SwAV提出用"交换预测"（swapped prediction）取代直接的特征对比：对同一张图片的两个不同增强视图，分别把它们的特征映射到一组可学习的原型向量上得到聚类编码（code），然后让一个视图的特征去预测另一个视图的编码，而不是直接比较两个特征向量。编码通过加了等分约束的最优传输问题在线（online）求解，避免了离线聚类需要遍历全部数据集的弊端，也不需要大型内存库或动量编码器。此外，论文还提出了多裁剪（multi-crop）数据增强策略，用多张小分辨率裁剪替代两张大图来增加视图数量，在几乎不增加计算量的前提下显著提升性能。最终SwAV在ImageNet线性评估上以标准ResNet-50达到75.3%的top-1精度，并在多个迁移任务上首次以冻结特征+线性分类器的方式超越了有监督预训练，证明自监督表征已具备与监督学习竞争的实用价值。
原文 arXiv:2006.09882;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.09882v5