Are All Layers Created Equal?
\nameChiyuan Zhang \AND\nameSamy Bengio \AND\nameYoram Singer
Abstract
Understanding deep neural networks is a major research objective with notable experimental and theoretical attention in recent years. The practical success of excessively large networks underscores the need for better theoretical analyses and justifications. In this paper we focus on layer-wise functional structure and behavior in overparameterized deep models. To do so, we study empirically the layers’ robustness to post-training re-initialization and re-randomization of the parameters. We provide experimental results which give evidence for the heterogeneity of layers. Morally, layers of large deep neural networks can be categorized as either “robust” or “critical”. Resetting the robust layers to their initial values does not result in adverse decline in performance. In many cases, robust layers hardly change throughout training. In contrast, re-initializing critical layers vastly degrades the performance of the network with test error essentially dropping to random guesses. Our study provides further evidence that mere parameter counting or norm calculations are too coarse in studying generalization of deep models, and “flatness” and robustness analysis of trained models need to
中文速览
过度参数化的深度神经网络在实际中表现出色，但人们对它为何能泛化仍缺乏清晰解释。研究者通过一个简单实验来剖析这一问题：训练完成后，逐层把参数"重置"回初始随机值或重新随机采样，观察模型性能如何变化，从而判断每一层对最终预测究竟有多重要。实验横跨全连接网络、卷积网络和残差网络，覆盖MNIST、CIFAR-10和ImageNet等主流图像分类任务，结果一致发现：网络各层呈现出明显的异质性——少数"关键层"（critical layers）一旦被重置，测试准确率就会跌至随机猜测水平；而大多数"鲁棒层"（robust layers）被重置后性能几乎不受影响，且这些层在训练过程中本身就几乎没有发生变化。这一发现说明，梯度训练会让过参数化网络"自我约束"，只让少量层真正参与表示学习，单纯靠参数量或范数来衡量模型复杂度和泛化能力是远远不够的，必须结合具体的网络架构来做更精细的分析。
原文 arXiv:1902.01996;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.01996v4