Scaling description of generalization with number of parameters in deep learning
Mario Geiger Affiliation: Institute of Physics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland Arthur Jacot Affiliation: Institute of Mathematics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland Stefano Spigler Affiliation: Institute of Physics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland Franck Gabriel Affiliation: Institute of Mathematics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland Levent Sagun Affiliation: Institute of Physics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland Stéphane d’Ascoli Affiliation: Laboratoire de Physique Statistique, École Normale Supérieure, PSL Research University, 75005 Paris, France Giulio Biroli Affiliation: Laboratoire de Physique Statistique, École Normale Supérieure, PSL Research University, 75005 Paris, France Clément Hongler Affiliation: Institute of Mathematics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland Matthieu Wyart Affiliation: Institute of Physics, École Polytechnique Fédérale de Lausanne, 1015 Lausanne, Switzerland
Abstract
Supervised deep learning involves the training of neural networks with a large number $N$ of parameters. For large enough $N$ , in the so-called over-parametrized regime, one can essentially fit the training data points. Sparsity-based arguments would suggest that the generalization error increases as $N$ grows past a certain threshold $N^{*}$ . Instead, empirical studies have shown that in the over-parametrized regime, generalization error keeps decreasing with $N$ . We resolve this paradox through a new framework. We rely on the so-called Neural Tangent Kernel, which connects large neural nets to kernel methods, to show that the initialization causes finite-size random fluctuations $\|f_{N}-\bar{f}_{N}\|\sim N^{-1/4}$ of the neural net output function $f_{N}$ around its expectation $\bar{f}_{N}$ . These affect the generalization error $\epsilon_{N}$ for classification: under natural assumptions, it decays to a plateau value $\epsilon_{\infty}$ in a power-law fashion $\sim N^{-1/2}$ . This description breaks down at a so-called jamming transition $N=N^{*}$ . At this threshold, we argue that $\|f_{N}\|$ diverges. This result leads to a plausible explanation for the cusp in test err
原文 arXiv:1901.01608;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.01608v5