Exploring the Equivalence of Siamese Self-Supervised Learning via A Unified Gradient Framework
Chenxin Tao1∗†, Honghui Wang1∗†, Xizhou Zhu2∗, Jiahua Dong3∗†, Shiji Song1, Gao Huang1,4, Jifeng Dai2,4🖂 1Tsinghua University, 2SenseTime Research, 3Zhejiang University 4Beijing Academy of Artificial Intelligence, Beijing, China {tcx20, {zhuwalter,
Abstract
Self-supervised learning has shown its great potential to extract powerful visual representations without human annotations. Various works are proposed to deal with self-supervised learning from different perspectives: (1) contrastive learning methods (e.g., MoCo, SimCLR) utilize both positive and negative samples to guide the training direction; (2) asymmetric network methods (e.g., BYOL, SimSiam) get rid of negative samples via the introduction of a predictor network and the stop-gradient operation; (3) feature decorrelation methods (e.g., Barlow Twins, VICReg) instead aim to reduce the redundancy between feature dimensions. These methods appear to be quite different in the designed loss functions from various motivations. The final accuracy numbers also vary, where different networks and tricks are utilized in different works. In this work, we demonstrate that these methods can be unified into the same form. Instead of comparing their loss functions, we derive a unified formula through gradient analysis. Furthermore, we conduct fair and detailed experiments to compare their performances. It turns out that there is little gap between these methods, and the use of momentum encoder
中文速览
自监督视觉表示学习领域存在对比学习、非对称网络、特征去冗余三大类方法，它们损失函数形式各异，训练技巧也不统一，很难直接比较、也难以理解各自奏效的根本原因。本文绕开损失函数，从梯度角度出发，证明这三类方法在反向传播时都遵循同一结构：梯度由"正样本梯度"（拉近同图两视角）和"负样本梯度"（推开不同图样本的加权组合）两部分构成，差别仅在于目标分支类型和负样本集合的构建方式。在这一统一框架下，作者进行了公平受控的对比实验，发现各方法本身性能差距很小，动量编码器（momentum encoder）才是影响最终效果的关键因素。基于此，作者提出了简洁高效的梯度范式 UniGrad——无需记忆库或非对称预测器网络，即可在线性评估、半监督学习及多项下游视觉任务上达到当前最优水平，为统一理解和设计自监督学习方法提供了清晰的理论基础。
原文 arXiv:2112.05141;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.05141v3