Scaling Laws vs Model Architectures: How does Inductive Bias Influence Scaling?
Yi Tay Mostafa Dehghani∗ Samira Abnar Hyung Won Chung William Fedus Jinfeng Rao Sharan Narang Vinh Q. Tran Dani Yogatama† Donald Metzler Google Research、DeepMind† Yi and Mostafa contributed equally. Samira is now at Apple.
Abstract
There have been a lot of interest in the scaling properties of Transformer models (Kaplan et al., 2020). However, not much has been done on the front of investigating the effect of scaling properties of different inductive biases and model architectures. Do model architectures scale differently? If so, how does inductive bias affect scaling behaviour? How does this influence upstream (pretraining) and downstream (transfer)? This paper conducts a systematic study of scaling behaviour of ten diverse model architectures such as Transformers, Switch Transformers, Universal Transformers, Dynamic convolutions, Performers, and recently proposed MLP-Mixers. Via extensive experiments, we show that (1) architecture is an indeed an important consideration when performing scaling and (2) the best performing model can fluctuate at different scales. We believe that the findings outlined in this work has significant implications to how model architectures are currently evaluated in the community.
中文速览
不同神经网络架构在扩大规模时的表现差异悬殊，但学界此前对此缺乏系统认识。研究者对十种架构（包括标准Transformer、Switch Transformer、Universal Transformer、Performer、MLP-Mixer等）从1500万到400亿参数进行了大规模预训练与微调实验，系统测量各架构的扩展规律（scaling laws）。结果发现，许多架构在小规模时与标准Transformer旗鼓相当甚至更优，但随着算力增加却无法持续改进，而标准Transformer虽然在单点性能上并非总是最佳，却展现出最稳健的扩展系数；此外，线性注意力模型（如Performer）的扩展能力尤为令人担忧，预训练困惑度几乎没有随规模提升而下降。这项研究的重要意义在于：学界长期依赖"单一规模下的点式对比"来评价新架构，但一个模型在某个计算区间的优势并不能保证它在更大规模下仍具竞争力，也不能保证预训练性能的提升会迁移到下游任务，这对未来模型设计和评测标准都有深远影响。
原文 arXiv:2207.10551;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2207.10551v1