Unified Scaling Laws for Routed Language Models
Aidan Clark∗, Diego de las Casas∗, Aurelia Guy∗, Arthur Mensch∗ Michela Paganini, Jordan Hoffmann, Bogdan Damoc, Blake Hechtman‡, Trevor Cai, Sebastian Borgeaud, George van den Driessche, Eliza Rutherford, Tom Hennigan, Matthew Johnson‡, Katie Millican, Albin Cassirer, Chris Jones, Elena Buchatskaya, David Budden, Laurent Sifre, Simon Osindero, Oriol Vinyals, Jack Rae, Erich Elsen, Koray Kavukcuoglu, Karen Simonyan DeepMind Google Research‡
Abstract
The performance of a language model has been shown to be effectively modeled as a power-law in its parameter count. Here we study the scaling behaviors of Routing Networks: architectures that conditionally use only a subset of their parameters while processing an input. For these models, parameter count and computational requirement form two independent axes along which an increase leads to better performance. In this work we derive and justify scaling laws defined on these two variables which generalize those known for standard language models and describe the performance of a wide range of routing architectures trained via three different techniques. Afterwards we provide two applications of these laws: first deriving an Effective Parameter Count along which all models scale at the same rate, and then using the scaling coefficients to give a quantitative comparison of the three routing techniques considered. Our analysis derives from an extensive evaluation of Routing Networks across five orders of magnitude of size, including models with hundreds of experts and hundreds of billions of parameters.
中文速览
路由网络（Routing Networks）可以在不显著增加计算量的前提下大幅扩充模型参数规模，但此前人们并不清楚这类模型的性能究竟如何随参数量和计算量这两个维度协同变化。研究者系统训练了横跨五个数量级、最大达2000亿参数的路由语言模型，覆盖三种截然不同的路由训练方式——基于最优传输的稀疏混合专家（Sinkhorn-BASE）、固定哈希路由和强化学习路由——并为所有这些模型推导出一套统一的双变量幂律缩放定律，同时将其推广为以推理计算量和总参数量为轴的更通用形式。基于这套定律，他们进一步定义了"等效参数量"（Effective Parameter Count），使稠密模型与路由模型的性能可以在同一坐标轴上直接比较，并对三种路由技术的效率给出了量化排名。这项工作不仅填补了路由网络缩放规律的空白，也为在算力有限条件下合理选择和设计大规模稀疏模型提供了可操作的理论依据。
原文 arXiv:2202.01169;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2202.01169v2