Gemma 2: Improving Open Language Models at a Practical Size
Gemma Team Google DeepMind\auth
Abstract
In this work, we introduce Gemma 2, a new addition to the Gemma family of lightweight, state-of-the-art open models, ranging in scale from 2 billion to 27 billion parameters. In this new version, we apply several known technical modifications to the Transformer architecture, such as interleaving local-global attentions (Beltagy et al., 2020a) and group-query attention (Ainslie et al., 2023). We also train the 2B and 9B models with knowledge distillation (Hinton et al., 2015) instead of next token prediction. The resulting models deliver the best performance for their size, and even offer competitive alternatives to models that are 2-3 $\times$ bigger. We release all our models to the community.
中文速览
谷歌推出了 Gemma 2 系列轻量级开放大语言模型，参数规模覆盖 20 亿到 270 亿，核心目标是让小模型在不单纯堆砌训练数据的前提下大幅提升性能。关键手段是用知识蒸馏（knowledge distillation）替代传统的下一词预测训练：让一个大模型充当"教师"，把它对每个词的概率分布传递给较小的"学生"模型，相当于用更丰富的学习信号来弥补数据量的不足，2B 和 9B 模型的训练量超过理论最优计算量的 50 倍。结合局部-全局交替注意力机制和分组查询注意力（Grouped-Query Attention）等架构改进，Gemma 2 在同等规模开放模型中达到最优，甚至能与体量大两三倍的模型一较高下。这对开源社区意义重大，因为它证明了在算力和数据受限的情况下，通过蒸馏等训练策略可以显著突破小模型的性能天花板，让更多开发者以更低成本获得高质量模型。
原文 arXiv:2408.00118;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2408.00118v3