Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics
Vinay Ramasesh Google Mountain View, CA Ethan Dyer Google Mountain View, CA Maithra Raghu Google Mountain View, CA
Abstract
A central challenge in developing versatile machine learning systems is catastrophic forgetting — a model trained on tasks in sequence will suffer significant performance drops on earlier tasks. Despite the ubiquity of catastrophic forgetting, there is limited understanding of the underlying process and its causes. In this paper, we address this important knowledge gap, investigating how forgetting affects representations in neural network models. Through representational analysis techniques, we find that deeper layers are disproportionately the source of forgetting. Supporting this, a study of methods to mitigate forgetting illustrates that they act to stabilize deeper layers. These insights enable the development of an analytic argument and empirical picture relating the degree of forgetting to representational similarity between tasks. Consistent with this picture, we observe maximal forgetting occurs for task sequences with intermediate similarity. We perform empirical studies on the standard split CIFAR-10 setup and also introduce a novel CIFAR-100 based task approximating realistic input distribution shift.
中文速览
灾难性遗忘（catastrophic forgetting）是让神经网络按顺序学习多个任务时、模型在新任务上训练后严重遗忘旧任务知识的核心难题，但其内在机制一直缺乏系统性理解。研究者借助表征相似度分析（centered kernel alignment，CKA）、层冻结和层重置等工具，对多种网络架构在 split CIFAR-10 及他们新设计的 CIFAR-100 分布偏移任务上展开了细致实验，发现靠近输出端的深层（higher layers）才是遗忘的主要来源——这些层的表征在训练新任务时发生了剧烈变化，而浅层则相对稳定。进一步研究两类主流缓解方法（弹性权重巩固 EWC 和经验回放 replay buffer）时发现，尽管两者设计思路截然不同，其有效性都源于在新任务训练过程中稳定了深层表征。此外，论文还揭示了一个反直觉规律：遗忘最严重的并不发生在任务高度相似或高度不同时，而是在任务处于"中等相似度"时达到峰值，并通过一个解析框架从数学上给出了严格解释。这些发现不仅统一了此前看似矛盾的实验现象，也为未来针对性地设计更鲁棒的持续学习算法提供了清晰的理论抓手。
原文 arXiv:2007.07400;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.07400v1