GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Nan Du Yanping Huang Andrew M. Dai Simon Tong Dmitry Lepikhin Yuanzhong Xu Maxim Krikun Yanqi Zhou Adams Wei Yu Orhan Firat Barret Zoph Liam Fedus Maarten Bosma Zongwei Zhou Tao Wang Yu Emma Wang Kellie Webster Marie Pellat Kevin Robinson Kathleen Meier-Hellstern Toju Duke Lucas Dixon Kun Zhang Quoc V Le Yonghui Wu Zhifeng Chen Claire Cui
Abstract
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero, one and few-shot performance across 29 NLP tasks.
中文速览
万亿参数级别的语言模型虽然效果出色，但训练成本极高、能耗惊人，如何在保持性能的同时大幅降低计算开销，是大模型研究的核心挑战。谷歌提出了 GLaM（通才语言模型，Generalist Language Model），采用稀疏激活的专家混合（Mixture-of-Experts，MoE）架构——模型总参数高达 1.2 万亿，但处理每个词元时只激活约 970 亿个参数（约占总量的 8%），从而在推理时的计算量仅相当于一个规模小得多的稠密模型。在 29 项自然语言处理基准测试的零样本、单样本和小样本学习中，GLaM 的整体表现优于参数量仅 1750 亿的 GPT-3，训练能耗却只有 GPT-3 的三分之一，推理所需浮点运算量也减少了一半。这项工作首次在大规模小样本上下文学习场景中证明：稀疏解码器模型可以比同等计算量的稠密模型更强，为未来以更低能耗训练高质量大模型指出了一条切实可行的路径。
原文 arXiv:2112.06905;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.06905v2