Primer: Searching for Efficient Transformers for Language Modeling
David R. So, Wojciech Mańke, Hanxiao Liu, Zihang Dai, Noam Shazeer, Quoc V. Le Google Research, Brain Team {davidso, wojciechm, hanxiaol, zihangd, noam,
Abstract
Large Transformer models have been central to recent advances in natural language processing. The training and inference costs of these models, however, have grown rapidly and become prohibitively expensive. Here we aim to reduce the costs of Transformers by searching for a more efficient variant. Compared to previous approaches, our search is performed at a lower level, over the primitives that define a Transformer TensorFlow program. We identify an architecture, named Primer, that has a smaller training cost than the original Transformer and other variants for auto-regressive language modeling. Primer’s improvements can be mostly attributed to two simple modifications: squaring ReLU activations and adding a depthwise convolution layer after each Q, K, and V projection in self-attention.
中文速览
为了降低大规模Transformer语言模型的训练成本，研究者设计了一套基于TensorFlow底层原语的进化搜索方法，在比以往更细粒度的搜索空间中寻找更高效的模型结构，最终发现了一个名为Primer（PRIMitives searched transformER）的新架构。Primer相比原始Transformer的提升主要来自两处简单改动：将前馈网络中的ReLU激活函数替换为平方ReLU，以及在自注意力的Q、K、V投影之后各加入一层深度可分离卷积（depthwise convolution）。实验表明，在1.9B参数规模下，Primer仅需Transformer约三分之一的训练算力就能达到相同的单样本（one-shot）下游任务表现；在500M参数的T5配置上，Primer可将训练成本压缩约4倍。这项工作的意义在于：两处核心改动极为简单，无需重新设计整个训练流程，可以直接插入现有Transformer代码库并立即获得显著的效率收益，为日益高昂的大模型训练成本提供了一条低门槛的改进路径。
原文 arXiv:2109.08668;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2109.08668v2