One Epoch Is All You Need
Aran Komatsuzaki School of Mathematics Georgia Institute of Technology Atlanta, GA 30332, USA
Abstract
In unsupervised learning, collecting more data is not always a costly process unlike the training. For example, it is not hard to enlarge the 40GB WebText used for training GPT-2 by modifying its sampling methodology considering how many webpages there are in the Internet. On the other hand, given that training on this dataset already costs tens of thousands of dollars, training on a larger dataset naively is not cost-wise feasible. In this paper, we suggest to train on a larger dataset for only one epoch unlike the current practice, in which the unsupervised models are trained for from tens to hundreds of epochs. Furthermore, we suggest to adjust the model size and the number of iterations to be performed appropriately. We show that the performance of Transformer language model becomes dramatically improved in this way, especially if the original number of epochs is greater. For example, by replacing the training for 10 epochs with the one epoch training, this translates to 1.9-3.3x speedup in wall-clock time in our settings and more if the original number of epochs is greater. Under one epoch training, no overfitting occurs, and regularization method does nothing but slows down t
中文速览
大规模无监督语言模型（如GPT-2、BERT）的训练动辄耗资数万美元，但现有做法普遍把同一批数据反复训练几十乃至上百轮（epoch），这其实是一种严重浪费。本文提出：只需把数据集扩大、每个样本只看一遍（即"单轮训练"，one-epoch training），同时根据经验法则把模型参数量与迭代次数的比例调整到接近最优点，就能在不增加总计算量的前提下大幅提升语言模型性能。实验表明，单轮训练下过拟合完全消失、正则化只会拖慢训练，测试损失随迭代次数呈幂律下降；两项方法叠加后，在相同壁钟时间内可实现3.3–5.1倍的速度提升。这一发现意味着，训练BERT、GPT-2这类顶尖模型的成本有望大幅压缩——甚至可能降至原来的十分之一——为整个领域以更低代价探索更大规模模型提供了切实可行的路径。
原文 arXiv:1906.06669;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.06669v1