Transcending Scaling Laws with 0.1% Extra Compute
Yi Tay Jason Wei Hyung Won Chung Vinh Q. Tran David R. So Siamak Shakeri Xavier Garcia Huaixiu Steven Zheng Jinfeng Rao Aakanksha Chowdhery Denny Zhou Donald Metzler Slav Petrov Neil Houlsby Quoc V. Le Mostafa Dehghani Google
Abstract
Scaling language models improves performance but comes with significant computational costs. This paper proposes UL2R, a method that substantially improves existing language models and their scaling curves with a relatively tiny amount of extra compute. The key idea is to continue training a state-of-the-art large language model (e.g., PaLM) on a few more steps with UL2’s mixture-of-denoiser objective. We show that, with almost negligible extra computational costs and no new sources of data, we are able to substantially improve the scaling properties of large language models on downstream metrics. In this paper, we continue training PaLM with UL2R, introducing a new set of models at 8B, 62B, and 540B scale which we call U-PaLM. Impressively, at 540B scale, we show an approximately 2x computational savings rate where U-PaLM achieves the same performance as the final PaLM 540B model at around half its computational budget (i.e., saving $\sim$ 4.4 million TPUv4 hours).
中文速览
大规模语言模型越训越强，但代价是天文数字般的算力消耗；这项工作提出了一个叫 UL2R（UL2 Restore）的方法，核心思路是在已经训练好的因果语言模型（如 PaLM）基础上，用"混合去噪"目标（mixture-of-denoisers）——包括前缀语言建模和填空式预训练——再多训练一小段，额外计算量仅占原始训练成本的 0.1%–1%，且不需要任何新数据。研究者将这一方法应用于 8B、62B、540B 三个规模的 PaLM，得到新模型系列 U-PaLM，结果表明 540B 规模的 U-PaLM 在达到原版 PaLM 540B 最终性能时只需约一半的计算量，节省了约 440 万 TPUv4 小时。更值得关注的是，U-PaLM 在一批原本要到 540B 才出现"涌现能力"（emergent abilities）的 BIG-Bench 难题上，62B 甚至 8B 就能展现出类似能力，说明丰富的预训练目标本身就能在不增加模型参数的前提下解锁新能力，为如何更高效地提升超大模型性能提供了一条实用新路径。
原文 arXiv:2210.11399;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.11399v2