Online Structured Laplace Approximations For Overcoming Catastrophic Forgetting
Hippolyt Ritter University College London、Aleksandar Botev University College London、David Barber University College London、Alan Turing Institute Corresponding author:
Abstract
We introduce the Kronecker factored online Laplace approximation for overcoming catastrophic forgetting in neural networks. The method is grounded in a Bayesian online learning framework, where we recursively approximate the posterior after every task with a Gaussian, leading to a quadratic penalty on changes to the weights. The Laplace approximation requires calculating the Hessian around a mode, which is typically intractable for modern architectures. In order to make our method scalable, we leverage recent block-diagonal Kronecker factored approximations to the curvature. Our algorithm achieves over $90\%$ test accuracy across a sequence of $50$ instantiations of the permuted MNIST dataset, substantially outperforming related methods for overcoming catastrophic forgetting.
中文速览
神经网络在依次学习多个任务时，往往会把旧任务的知识"灾难性遗忘"——新任务的训练会把权重推离旧任务的最优解。本文借助贝叶斯在线学习的思路，每完成一个任务就用拉普拉斯近似（Laplace approximation）把当前参数后验拟合成一个高斯分布，再把这个高斯当作下一个任务的先验，从而将"不要忘记旧知识"自然地转化为对权重变化的二次惩罚项。为了让这套方法在现代大型网络上可行，作者引入了克罗内克分解（Kronecker factored approximation）来高效近似海森矩阵，能够捕捉同一层内权重之间的交互，而不是像以往方法那样只用对角近似。在连续50个排列MNIST任务的测试中，该方法测试准确率超过90%，显著优于弹性权重巩固（EWC）等同类基线，证明更精确的曲率近似对克服灾难性遗忘至关重要。
原文 arXiv:1805.07810;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1805.07810v1