Every Model Learned by Gradient Descent Is Approximately a Kernel Machine
\namePedro Domingos \addrPaul G. Allen School of Computer Science、Engineering University of Washington Seattle, WA 98195-2350, USA
Abstract
Deep learning’s successes are often attributed to its ability to automatically discover new representations of the data, rather than relying on handcrafted features like other learning methods. We show, however, that deep networks learned by the standard gradient descent algorithm are in fact mathematically approximately equivalent to kernel machines, a learning method that simply memorizes the data and uses it directly for prediction via a similarity function (the kernel). This greatly enhances the interpretability of deep network weights, by elucidating that they are effectively a superposition of the training examples. The network architecture incorporates knowledge of the target function into the kernel. This improved understanding should lead to better learning algorithms.
中文速览
深度学习长期被认为能够从数据中自动发现新的表示，但这篇论文从数学上证明：凡是用标准梯度下降算法训练的模型，无论网络结构多复杂，其最终结果都近似等价于一种核机器（kernel machine）——也就是通过一个"路径核（path kernel）"把训练样本直接记忆下来、再用相似度来做预测。具体来说，网络的权重可以被解读为训练样本在梯度空间中的叠加，而网络架构本身则决定了这个核的形状，从而将领域知识隐式地编码进去。这一发现一方面大幅提升了深度网络的可解释性，另一方面也挑战了"深度学习靠自动表示学习取胜"的主流叙事，揭示梯度下降本质上只是在一个预定义的特征空间中做选择。这项工作不仅为理解深度网络的泛化、脆弱性和"记忆"现象提供了统一解释，还为把深度学习的结构知识直接移植到核方法、以及探索超越梯度下降的表示学习算法指明了方向。
原文 arXiv:2012.00152;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.00152v1