GLaM: Efficient Scaling of Language Models with Mixture-of-Experts
Nan Du Affiliation: Google Yanping Huang Affiliation: Google Andrew M. Dai Affiliation: Google Simon Tong Affiliation: Google Dmitry Lepikhin Affiliation: Google Yuanzhong Xu Affiliation: Google Maxim Krikun Affiliation: Google Yanqi Zhou Affiliation: Google Adams Wei Yu Affiliation: Google Orhan Firat Affiliation: Google Barret Zoph Affiliation: Google Liam Fedus Affiliation: Google Maarten Bosma Affiliation: Google Zongwei Zhou Affiliation: Google Tao Wang Affiliation: Google Yu Emma Wang Affiliation: Google Kellie Webster Affiliation: Google Marie Pellat Affiliation: Google Kevin Robinson Affiliation: Google Kathleen Meier-Hellstern Affiliation: Google Toju Duke Affiliation: Google Lucas Dixon Affiliation: Google Kun Zhang Affiliation: Google Quoc V Le Affiliation: Google Yonghui Wu Affiliation: Google Zhifeng Chen Affiliation: Google Claire Cui Affiliation: Google
Abstract
Scaling language models with more data, compute and parameters has driven significant progress in natural language processing. For example, thanks to scaling, GPT-3 was able to achieve strong results on in-context learning tasks. However, training these large dense models requires significant amounts of computing resources. In this paper, we propose and develop a family of language models named GLaM (Generalist Language Model), which uses a sparsely activated mixture-of-experts architecture to scale the model capacity while also incurring substantially less training cost compared to dense variants. The largest GLaM has 1.2 trillion parameters, which is approximately 7x larger than GPT-3. It consumes only 1/3 of the energy used to train GPT-3 and requires half of the computation flops for inference, while still achieving better overall zero, one and few-shot performance across 29 NLP tasks.
原文 arXiv:2112.06905;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.06905v2