LoCo: Local Contrastive Representation Learning
Yuwen Xiong Uber ATG University of Toronto、Mengye Ren Uber ATG University of Toronto、Raquel Urtasun Uber ATG University of Toronto
Abstract
Deep neural nets typically perform end-to-end backpropagation to learn the weights, a procedure that creates synchronization constraints in the weight update step across layers and is not biologically plausible. Recent advances in unsupervised contrastive representation learning invite the question of whether a learning algorithm can also be made local, that is, the updates of lower layers do not directly depend on the computation of upper layers. While Greedy InfoMax [39] separately learns each block with a local objective, we found that it consistently hurts readout accuracy in state-of-the-art unsupervised contrastive learning algorithms, possibly due to the greedy objective as well as gradient isolation. In this work, we discover that by overlapping local blocks stacking on top of each other, we effectively increase the decoder depth and allow upper blocks to implicitly send feedbacks to lower blocks. This simple design closes the performance gap between local learning and end-to-end contrastive learning algorithms for the first time. Aside from standard ImageNet experiments, we also show results on complex downstream tasks such as object detection and instance segmentation dir
中文速览
端到端反向传播训练深度神经网络时，每一层的参数更新都依赖来自顶层的梯度，这不仅造成层间同步等待、消耗大量显存，也与大脑局部学习的生物机制相悖。已有工作 Greedy InfoMax 尝试给每个网络模块单独加一个对比学习损失并切断层间梯度，实现"局部学习"，但作者发现这种做法在 ImageNet 等大规模数据集上会显著损害表征质量，根本原因在于低层模块无法获得来自高层的反馈、且各模块的解码器感受野太浅。为此，本文提出 LoCo 方法：将相邻两个网络阶段组成一个局部训练单元，中间那个阶段同时被上下两个单元共享，从而在不破坏局部性的前提下，让上层的梯度信号通过"桥接阶段"隐式传回下层，同时等效地加深了各局部单元的解码器深度。在 ImageNet 表征学习基准以及 MS-COCO 目标检测、实例分割等下游任务上，LoCo 的性能首次与端到端对比学习方法持平，这意味着局部学习有望在不牺牲精度的情况下大幅降低训练显存占用并天然支持模型并行，为更高效、更具生物合理性的深度学习训练范式提供了切实可行的路径。
原文 arXiv:2008.01342;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2008.01342v2