The Llama 3 Herd of Models
Llama Team AI @ Meta [
Abstract
Modern artificial intelligence (AI) systems are powered by foundation models. This paper presents a new set of foundation models, called Llama 3. It is a herd of language models that natively support multilinguality, coding, reasoning, and tool usage. Our largest model is a dense Transformer with 405B parameters and a context window of up to 128K tokens. This paper presents an extensive empirical evaluation of Llama 3. We find that Llama 3 delivers comparable quality to leading language models such as GPT-4 on a plethora of tasks. We publicly release Llama 3, including pre-trained and post-trained versions of the 405B parameter language model and our Llama Guard 3 model for input and output safety. The paper also presents the results of experiments in which we integrate image, video, and speech capabilities into Llama 3 via a compositional approach. We observe this approach performs competitively with the state-of-the-art on image, video, and speech recognition tasks. The resulting models are not yet being broadly released as they are still under development.
中文速览
Meta 推出的 Llama 3 系列旨在打造能媲美 GPT-4 的开源基础大语言模型，核心挑战是如何在数据质量、训练规模和系统复杂度之间取得最优平衡。研究团队在约 15 万亿多语言 token 上预训练了 8B、70B、405B 三档稠密 Transformer 模型，随后通过监督微调（SFT）、拒绝采样（RS）和直接偏好优化（DPO）完成后训练，使模型具备多语言、代码、推理和工具调用能力，并额外引入图像、视频、语音模块进行多模态扩展实验。大量基准测试和人工评估表明，405B 旗舰模型在多项任务上与 GPT-4 持平，较小的 8B 和 70B 模型也在同参数量级中领先，同时安全性相比前代 Llama 2 显著改善。Meta 全面开放了预训练和后训练权重，这对学术界和开源社区意义重大，标志着顶级性能的大模型首次以完全公开的方式向所有人提供。
原文 arXiv:2407.21783;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2407.21783v3