Representational Continuity for Unsupervised Continual Learning
Divyam Madaan1 Jaehong Yoon2,3†23†{}^{2,3\leavevmode\nobreak\ \color[rgb]{0.6,0,0}{\dagger}} Yuanchun Li5,6 Yunxin Liu5,6 Sung Ju Hwang2,4 New York University1 KAIST2 Microsoft Research3 AITRICS4 Institute for AI Industry Research (AIR)5 Tsinghua University6 Corresponding author. † The work was done while the author was an intern at Microsoft Research.
Abstract
Continual learning (CL) aims to learn a sequence of tasks without forgetting the previously acquired knowledge. However, recent CL advances are restricted to supervised continual learning (SCL) scenarios. Consequently, they are not scalable to real-world applications where the data distribution is often biased and unannotated. In this work, we focus on unsupervised continual learning (UCL), where we learn the feature representations on an unlabelled sequence of tasks and show that reliance on annotated data is not necessary for continual learning. We conduct a systematic study analyzing the learned feature representations and show that unsupervised visual representations are surprisingly more robust to catastrophic forgetting, consistently achieve better performance, and generalize better to out-of-distribution tasks than SCL. Furthermore, we find that UCL achieves a smoother loss landscape through qualitative analysis of the learned representations and learns meaningful feature representations. Additionally, we propose Lifelong Unsupervised Mixup (Lump), a simple yet effective technique that interpolates between the current task and previous tasks’ instances to alleviate catastrop
中文速览
持续学习（Continual Learning）领域长期以来几乎只关注有标签数据的监督场景，但现实中数据往往是无标签且分布不断变化的。这项工作系统研究了无监督持续学习（Unsupervised Continual Learning, UCL）——让模型在一系列无标签任务上依次学习特征表示，同时尽量不遗忘旧知识。研究发现，基于SimSiam、BarlowTwins等自监督方法学到的无监督表示，比监督持续学习对"灾难性遗忘"（catastrophic forgetting）更鲁棒，在分布外任务和少样本场景下也表现更好，其原因在于无监督模型学到的是更通用的特征，损失曲面也更平坦光滑。在此基础上，作者提出了一种简单有效的方法——终身无监督混合（Lifelong Unsupervised Mixup, Lump），通过将当前任务样本与回放缓冲区中的历史样本做插值混合来训练模型，无需额外超参数即可进一步显著降低遗忘，在多个基准上超越了现有有监督持续学习方法，为将自监督表示学习扩展到真实世界的持续变化场景提供了有力支撑。
原文 arXiv:2110.06976;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.06976v3