Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets
Julia Kreutzera,b, Isaac Caswella, Lisa Wanga, Ahsan Wahabc, Daan van Escha, Nasanbayar Ulzii-Orshikhd, Allahsera Tapob,e, Nishant Subramanib,δ, Artem Sokolova, Claytone Sikasoteb,g, Monang Setyawanh, Supheakmungkol Sarinh, Sokhar Sambb,i, Benoît Sagotj, Clara Riveraa, Annette Riosk, Isabel Papadimitrioul, Salomey Oseib,m, Pedro Ortiz Suarezj,n, Iroro Orifeb,o, Kelechi Oguejib,p, Andre Niyongabo Rubungob,q, Toan Q. Nguyenr, Mathias Müllerk, André Müllerk, Shamsuddeen Hassan Muhammadb,s, Nanda Muhammadh, Ayanda Mnyakenih, Jamshidbek Mirzakhalovc,t, Tapiwanashe Matangirah, Colin Leongb, Nze Lawsonh, Sneha Kuduguntaa, Yacine Jerniteb,u, Mathias Jennyk, Orhan Firata,c, Bonaventure F. P. Dossoub,v, Sakhile Dlaminih, Nisansa de Silvaw, Sakine Çabuk Ballık, Stella Bidermanx, Alessia Battistik, Ahmed Baruwab,y, Ankur Bapnaa, Pallavi Baljekara, Israel Abebe Azimeb,i, Ayodele Awokoyab,z, Duygu Atamanc,k, Orevaoghene Ahiab,α, Oghenefego Ahiah, Sweta Agrawalβ, Mofetoluwa Adeyemib,γ, aGoogle Research, bMasakhane NLP, cTurkic Interlingua, dHaverford College, eRobotsMali, fIntel Labs, gUniversity of Zambia, hGoogle, iAIMS-AMMI, jInria, kUniversity of Zurich, lStanford University, mKwame Nkrumah University of Science and Technology, nSorbonne Université, oNiger-Volta LTI, pUniversity of Waterloo qUniversity of Electronic Science and Technology of China, rUniversity of Notre Dame, sBayero University Kano, tUniversity of South Florida, uHugging Face, vJacobs University Bremen, wUniversity of Moratuwa, xEleutherAI, yObafemi Awolowo University, zUniversity of Ibadan, αInstadeep, βUniversity of Maryland, γDefence Space Administration Abuja, δAllen Institute for Artificial Intelligence
Abstract
With the success of large-scale pre-training and multilingual modeling in Natural Language Processing (NLP), recent years have seen a proliferation of large, web-mined text datasets covering hundreds of languages. We manually audit the quality of 205 language-specific corpora released with five major public datasets (CCAligned, ParaCrawl, WikiMatrix, OSCAR, mC4). Lower-resource corpora have systematic issues: At least 15 corpora have no usable text, and a significant fraction contains less than 50% sentences of acceptable quality. In addition, many are mislabeled or use nonstandard/ambiguous language codes. We demonstrate that these issues are easy to detect even for non-proficient speakers, and supplement the human audit with automatic analyses. Finally, we recommend techniques to evaluate and improve multilingual corpora and discuss potential risks that come with low-quality data releases.
中文速览
大规模多语言网络爬取数据集是当前自然语言处理研究的重要基础，但这些数据对低资源语言究竟有多大用处，此前几乎无人仔细检验。研究团队招募了51位志愿者，对CCAligned、ParaCrawl、WikiMatrix、OSCAR、mC4五个主流公开数据集中的205个语言子集逐句进行人工审查，建立了一套错误分类体系，将问题归纳为语言错误、非语言内容、翻译错误等类别。审查结果触目惊心：至少15个语言子集几乎没有可用文本，相当一部分语言子集中质量合格的句子不足50%，且存在大量语言代码标注错误或使用不规范标识符的问题。这项工作揭示了当前多语言数据集在低资源语言上系统性的质量缺陷，并提出了一套低成本的审查方法和改进建议，对依赖这些数据训练模型的研究者和开发者具有重要的警示意义。
原文 arXiv:2103.12028;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.12028v4