CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data
Abstract
Pre-training text representations have led to significant improvements in many areas of natural language processing. The quality of these models benefits greatly from the size of the pretraining corpora as long as its quality is preserved. In this paper, we describe an automatic pipeline to extract massive high-quality monolingual datasets from Common Crawl for a variety of languages. Our pipeline follows the data processing introduced in fastText [Mikolov et al., 2017, Grave et al., 2018], that deduplicates documents and identifies their language. We augment this pipeline with a filtering step to select documents that are close to high quality corpora like Wikipedia. Keywords: Common Crawl, web data
中文速览
训练大规模语言模型离不开海量高质量文本，但如何从嘈杂的网络爬虫数据中高效筛选出干净的多语言语料，一直是个难题。研究者提出了CCNet这条自动化处理流程：先对Common Crawl网页数据做段落级去重和语言识别，再用维基百科训练的n-gram语言模型（KenLM）给每篇文档打困惑度（perplexity）分数，困惑度越低说明文档越接近高质量来源，从而把语料三等分并保留高质量部分。实验表明，仅处理2019年2月一个Common Crawl快照，就能得到174种语言共3.2TB的压缩文本，仅英语一种就有5320亿词元，是此前同类数据集的120倍；该流程在5000个CPU核上约9小时即可跑完一个快照。这项工作为低资源语言提供了远超现有维基百科规模的训练语料，且整套代码和预训练语言模型均已开源，对多语言NLP预训练研究具有重要的基础性价值。
原文 arXiv:1911.00359;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1911.00359v2