Efficient Large Scale Language Modeling with Mixtures of Experts
Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giri Anantharaman, Xian Li, Shuohui Chen, Halil Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Ves Stoyanov Meta AI Equal contribution. Authors listed alphabetically.
Abstract
Mixture of Experts layers (MoEs) enable efficient scaling of language models through conditional computation. This paper presents a detailed empirical study of how autoregressive MoE language models scale in comparison with dense models in a wide range of settings: in- and out-of-domain language modeling, zero- and few-shot priming, and full-shot fine-tuning. With the exception of fine-tuning, we find MoEs to be substantially more compute efficient. At more modest training budgets, MoEs can match the performance of dense models using $\sim$ 4 times less compute. This gap narrows at scale, but our largest MoE model (1.1T parameters) consistently outperforms a compute-equivalent dense model (6.7B parameters). Overall, this performance gap varies greatly across tasks and domains, suggesting that MoE and dense models generalize differently in ways that are worthy of future study. We make our code and models publicly available for research use.111https://github.com/pytorch/fairseq/tree/main/examples/moe_lm
中文速览
训练超大语言模型需要天文数字的算力，而"混合专家"（Mixture of Experts，MoE）架构通过让每个输入只激活模型的一小部分参数，可以在不成比例增加计算量的前提下大幅扩充模型容量。研究者系统比较了从数亿到逾万亿参数规模的MoE模型与同等计算量的稠密（dense）模型，覆盖了语言建模困惑度、零样本/少样本推理以及全量微调等多种评测场景。结果显示，在预训练预算较为有限时，MoE模型只需约四分之一的算力就能达到稠密模型的性能；即使扩展到最大规模，拥有1.1万亿参数的MoE模型仍持续优于计算量相当的67亿参数稠密模型——唯一的例外是全量微调场景，MoE在此表现相对逊色。这一发现意味着，对于算力受限的研究者而言，MoE是构建高性能大语言模型的极具潜力的替代路径，同时两类模型在不同任务和领域上呈现出的差异化泛化行为也值得深入探究。
原文 arXiv:2112.10684;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.10684v2