A Neural Scaling Law from the Dimension of the Data Manifold
Utkarsh Sharma、Jared Kaplan \ANDDepartment of Physics and Astronomy Johns Hopkins University
Abstract
When data is plentiful, the loss achieved by well-trained neural networks scales as a power-law $L\propto N^{-\alpha}$ in the number of network parameters $N$ . This empirical scaling law holds for a wide variety of data modalities, and may persist over many orders of magnitude. The scaling law can be explained if neural models are effectively just performing regression on a data manifold of intrinsic dimension $d$ . This simple theory predicts that the scaling exponents $\alpha\approx 4/d$ for cross-entropy and mean-squared error losses. We confirm the theory by independently measuring the intrinsic dimension and the scaling exponents in a teacher/student framework, where we can study a variety of $d$ and $\alpha$ by dialing the properties of random teacher networks. We also test the theory with CNN image classifiers on several datasets and with GPT-type language models.
中文速览
神经网络的损失随模型参数量增长呈幂律下降（L∝N^−α），但人们一直不清楚这个幂次α究竟由什么决定。这篇论文提出了一个简洁的理论解释：神经网络本质上是在对数据所在的低维流形（data manifold）做分段线性回归，流形的内在维度d越高，模型就需要指数级更多的参数才能提升精度，因此幂次α≈4/d。作者在师生网络（teacher/student）框架下通过独立测量α和d对理论进行了系统验证，还在CNN图像分类和GPT语言模型上做了测试，结果普遍支持这一预测，包括幂律成立范围的上限满足log N_max∝d等推论。这项工作把一个纯经验规律落实为可量化、可预测的几何图景，不仅解释了为何幂律能跨越多个数量级保持稳定、为何不同架构的α相近，还为通过降低数据流形维度来设计更高效架构提供了理论依据。
原文 arXiv:2004.10802;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.10802v1