Superposition of many models into one
Brian Cheung Redwood Center, BAIR UC Berkeley、Alex Terekhov Redwood Center UC Berkeley、Yubei Chen Redwood Center, BAIR UC Berkeley、Pulkit Agrawal BAIR UC Berkeley、Bruno Olshausen Redwood Center, BAIR UC Berkeley
Abstract
We present a method for storing multiple models within a single set of parameters. Models can coexist in superposition and still be retrieved individually. In experiments with neural networks, we show that a surprisingly large number of models can be effectively stored within a single parameter instance. Furthermore, each of these models can undergo thousands of training steps without significantly interfering with other models within the superposition. This approach may be viewed as the online complement of compression: rather than reducing the size of a network after training, we make use of the unrealized capacity of a network during training.
中文速览
神经网络其实存在大量冗余参数，训练完之后可以剪掉很多权重而不损失精度，但如何在训练阶段就充分利用这些"闲置容量"一直是个难题。这篇论文提出了"参数叠加"（Parameter Superposition，PSP）方法：通过给每个任务分配一个专属的"上下文"向量（如二值、复数或旋转矩阵），将多个模型的权重叠加编码进同一套参数，需要哪个模型时再用对应的上下文解码取出，类似傅里叶分析中用不同频率的正弦波叠加表示信号。实验表明，一套参数可以有效容纳数十个模型并分别训练数千步而互不干扰，在持续变化输入分布（如排列MNIST、旋转MNIST）和输出标签（如增量CIFAR）的场景下，PSP能大幅缓解灾难性遗忘，且每增加一个新任务只需额外存储极少量参数（最少仅需1个）。这项工作为内存受限环境下的在线多任务学习提供了一条全新思路，本质上是把网络压缩的逻辑反过来用——不是训练后压缩，而是训练时复用。
原文 arXiv:1902.05522;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.05522v2