Deep Learning Scaling is Predictable, Empirically
Joel Hestness, Sharan Narang, Newsha Ardalani, Gregory Diamos, Heewoo Jun,、Hassan Kianinejad, Md. Mostofa Ali Patwary, Yang Yang, Yanqi Zhou {joel,sharan,ardalaninewsha,gregdiamos,junheewoo,hassankianinejad, Baidu Research
Abstract
Deep learning (DL) creates impactful advances following a virtuous recipe: model architecture search, creating large training data sets, and scaling computation. It is widely believed that growing training sets and models should improve accuracy and result in better products. As DL application domains grow, we would like a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements to advance the state-of-the-art.
中文速览
大规模深度学习系统在实践中究竟需要多少数据、多大模型才能达到目标精度，目前缺乏可靠的预测方法。研究者们系统测量了机器翻译、语言模型、图像分类、语音识别四个领域共六种神经网络模型的"学习曲线"（learning curve）——即随训练数据量增加，泛化误差如何下降。结果发现，所有领域的泛化误差都遵循幂律（power-law）规律下降，但实测幂律指数普遍在 -0.07 到 -0.35 之间，远小于现有理论预测的 -0.5 或 -1，说明现有理论无法解释真实场景；此外，改进模型架构只能平移误差曲线的截距，并不能改变幂律指数本身，而所需模型参数量随数据量的增长也呈次线性幂律关系。这些发现意味着从业者可以用少量数据点拟合学习曲线，进而预测未来扩大数据集所能达到的精度上限，从而在数据采集、算力投入和系统规划上做出更有依据的决策，同时也为理论界指出了一个亟待解释的开放问题。
原文 arXiv:1712.00409;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1712.00409v1