Intriguing Properties of Randomly Weighted Networks: Generalizing While Learning Next to Nothing
Amir Rosenfeld John K.Tsotsos
Abstract
Training deep neural networks results in strong learned representations that show good generalization capabilities. In most cases, training involves iterative modification of all weights inside the network via back-propagation. In Extreme Learning Machines, it has been suggested to set the first layer of a network to fixed random values instead of learning it. In this paper, we propose to take this approach a step further and fix almost all layers of a deep convolutional neural network, allowing only a small portion of the weights to be learned. As our experiments show, fixing even the majority of the parameters of the network often results in performance which is on par with the performance of learning all of them. The implications of this intriguing property of deep neural networks are discussed and we suggest ways to harness it to create more robust representations.
中文速览
深度神经网络（deep neural network）通常需要通过反向传播更新全部参数才能达到好的性能，但实际上真的每个权重都必须被学习吗？研究者提出了一种"部分学习"的方法：在训练时将绝大多数层的权重固定为随机初始化值，只允许极小一部分权重被优化器更新。在多种主流网络架构（WRN、DenseNet、VGG、AlexNet）和图像分类数据集（CIFAR-10/100）上的大量实验表明，即便固定超过大半数的参数，网络性能的下降幅度出人意料地小，有时甚至与全量训练旗鼓相当。这一发现揭示了两个重要含义：当前流行的网络架构存在严重的过参数化（over-parameterization），而基于"网络学到了什么特征"的可解释性研究也可能没有我们想象的那么有意义；此外，这一特性还可以被利用来构建廉价的集成模型，只需固定主干权重、每次重新训练少量参数，便能以低代价获得多样性足够的多个模型，从而超越单一模型的表现。
原文 arXiv:1802.00844;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1802.00844v1