Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers
Yi Tay , Mostafa Dehghani∗, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama†, Ashish Vaswani, Donald Metzler Google Research、DeepMind† Equal contribution
Abstract
There remain many open questions pertaining to the scaling behaviour of Transformer architectures. These scaling decisions and findings can be critical, as training runs often come with an associated computational cost which have both financial and/or environmental impact. The goal of this paper is to present scaling insights from pretraining and finetuning Transformers. While Kaplan et al. (2020) presents a comprehensive study of the scaling behaviour of Transformer language models, the scope is only on the upstream (pretraining) loss. Therefore, it is still unclear if these set of findings transfer to downstream task within the context of the pretrain-finetune paradigm. The key findings of this paper are as follows: (1) we show that aside from only the model size, model shape matters for downstream fine-tuning, (2) scaling protocols operate differently at different compute regions, (3) widely adopted T5-base and T5-large sizes are Pareto-inefficient. To this end, we present improved scaling protocols whereby our redesigned models achieve similar downstream fine-tuning quality while having 50% fewer parameters and training 40% faster compared to the widely adopted T5-base model. W
中文速览
大规模训练Transformer模型的财务和环境成本不菲，如何在固定算力预算下设计最优模型结构，一直缺乏充分的实证指导——尤其是Kaplan等人的经典缩放定律只评估了预训练损失，从未验证能否迁移到下游微调任务。为此，研究者对超过200种T5架构配置（参数量从500万到300亿）进行了大规模预训练与微调实验，系统考察模型深度、宽度、注意力头数等形状参数对下游性能的影响。核心发现有三：下游微调性能对模型"形状"高度敏感，而非仅取决于参数量；预训练困惑度（perplexity）常常是下游质量的误导性指标；被广泛沿用的T5-Base和T5-Large尺寸在参数-性能帕累托前沿上并不高效。基于此，研究者提出"深窄（DeepNarrow）"缩放策略——增加层数、收窄宽度——在下游任务上取得与T5-Base相当甚至更好的效果，同时参数量减少50%、训练速度提升40%，为从业者在有限算力下做出更明智的模型设计决策提供了直接可用的实证依据。
原文 arXiv:2109.10686;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.10686v2