The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Only
Guilherme Penedo Quentin Malartic Daniel Hesslow Ruxandra Cojocaru Alessandro Cappelli Hamza Alobeidli Baptiste Pannier Ebtesam Almazrouei Julien Launay
Abstract
Large language models are commonly trained on a mixture of filtered web data and curated “high-quality” corpora, such as social media conversations, books, or technical papers. This curation process is believed to be necessary to produce performant models with broad zero-shot generalization abilities. However, as larger models requiring pretraining on trillions of tokens are considered, it is unclear how scalable is curation and whether we will run out of unique high-quality data soon. At variance with previous beliefs, we show that properly filtered and deduplicated web data alone can lead to powerful models; even significantly outperforming models from the state-of-the-art trained on The Pile. Despite extensive filtering, the high-quality data we extract from the web is still plentiful, and we are able to obtain five trillion tokens from CommonCrawl. We publicly release an extract of 600 billion tokens from our RefinedWeb dataset, and 1.3/7.5B parameters language models trained on it***Details about how to access Falcon LLM open source is available on falconllm.tii.ae.
中文速览
大规模语言模型的预训练一直依赖把网页数据与书籍、论文、社交媒体对话等"高质量"精选语料混合使用，业界普遍认为这种人工筛选不可或缺。这篇论文反驳了这一观点，提出只要对网页数据做好过滤与去重，单靠网页数据就能训练出强大的语言模型。研究团队构建了一套名为 MDR（MacroData Refinement）的数据处理流水线，从 CommonCrawl 中提取出高质量的 RefinedWeb 数据集，总规模达五万亿词元，并公开发布了其中六千亿词元的子集以及在此基础上训练的 1.3B 和 7.5B 参数模型。实验表明，仅用这份网页数据训练的模型在零样本基准测试上明显优于基于 The Pile 等精选语料库训练的同类模型，这不仅挑战了"网页数据质量天然低劣"的固有认知，也为缓解大模型训练对稀缺人工标注数据的依赖提供了一条可扩展的新路径。
原文 arXiv:2306.01116;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.01116v1