Training BatchNorm and Only BatchNorm: On the Expressive Power of Random Features in CNNs
Jonathan Frankle MIT CSAIL \AndDavid J. Schwab CUNY Graduate Center, ITS Facebook AI Research \AndAri S. Morcos Facebook AI Research Work done while an intern and student researcher at Facebook AI Research.
Abstract
A wide variety of deep learning techniques from style transfer to multitask learning rely on training affine transformations of features. Most prominent among these is the popular feature normalization technique BatchNorm, which normalizes activations and then subsequently applies a learned affine transform. In this paper, we aim to understand the role and expressive power of affine parameters used to transform features in this way. To isolate the contribution of these parameters from that of the learned features they transform, we investigate the performance achieved when training only these parameters in BatchNorm and freezing all weights at their random initializations. Doing so leads to surprisingly high performance considering the significant limitations that this style of training imposes. For example, sufficiently deep ResNets reach 82% (CIFAR-10) and 32% (ImageNet, top-5) accuracy in this configuration, far higher than when training an equivalent number of randomly chosen parameters elsewhere in the network. BatchNorm achieves this performance in part by naturally learning to disable around a third of the random features. Not only do these results highlight the expressive p
中文速览
冻结神经网络几乎全部权重、只训练BatchNorm里的缩放系数γ和偏置β，深层ResNet（如ResNet-86/200）在CIFAR-10上仍能达到82%准确率、在ImageNet上达到32%的Top-5准确率，而这两类参数加起来还不到全网络参数量的0.6%。研究者通过对比"随机选取等量参数进行训练"的基准实验发现，同等数量的随机参数只能达到56%（CIFAR-10）和4%（ImageNet），说明γ/β之所以高效，恰恰在于它们作为"逐特征缩放与平移"的特殊位置，而非仅仅是参数数量的贡献。进一步分析表明，γ会自然地将四分之一到一半的通道压缩至接近零，借助稀疏化来从随机特征中筛选有用信号。这项工作揭示了仿射参数（affine parameters）在深度学习中被低估的表达能力，同时也说明随机初始化的网络本身就蕴含着足够丰富的"原材料"，仅靠对随机特征做移位和缩放，就能构建出具有实用精度的图像分类器。
原文 arXiv:2003.00152;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.00152v3