When Do Neural Networks Outperform Kernel Methods?
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, Andrea Montanari Department of Electrical Engineering, Stanford UniversityDepartment of Statistics, University of California, BerkeleyDepartment of Statistics, Stanford UniversityGoogle Research, Brain Team
Abstract
For a certain scaling of the initialization of stochastic gradient descent (SGD), wide neural networks (NN) have been shown to be well approximated by reproducing kernel Hilbert space (RKHS) methods. Recent empirical work showed that, for some classification tasks, RKHS methods can replace NNs without a large loss in performance. On the other hand, two-layers NNs are known to encode richer smoothness classes than RKHS and we know of special examples for which SGD-trained NN provably outperform RKHS. This is true even in the wide network limit, for a different scaling of the initialization.
中文速览
神经网络（NN）和核方法（RKHS）到底谁更强，这个问题在学界一直有争议——有人用实验说两者差不多，也有人用理论证明NN在某些任务上碾压核方法。本文提出了一个"尖峰协变量模型"（spiked covariates model）来统一解释这两种看似矛盾的结论：当输入数据本身就集中在与目标函数相关的低维子空间上时（信噪比高），核方法的维数灾难会得到缓解，两者表现相近；而当数据分布接近各向同性时，核方法深受维数灾难拖累，NN凭借自动学习低维表示的能力则显著占优。作者用精确的理论刻画了这一权衡关系，并通过在Fashion-MNIST图像分类任务中向高频成分注入噪声（使数据更各向同性）来验证：这种扰动让核方法和随机特征模型的准确率大幅下滑，而NN受到的影响明显更小。这项工作的意义在于，它给出了一个清晰的判断准则——数据的低维结构是否与目标函数对齐，决定了在实际任务中应该选择NN还是核方法。
原文 arXiv:2006.13409;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2006.13409v2