Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus
Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna Google Research, 1600 Amphitheatre Parkway, Mountain View, CA 94043
Abstract
Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean se
中文速览
自动语言识别（Language Identification, LangID）一直被认为是个"已解决"的问题，但当真正用它从网络爬虫中为500多种低资源语言收集文本时，研究者发现许多语言的实际识别精确率只有5%左右——远比实验室测试集上的96%+准确率惨烈得多。问题根源是多方面的：网络上语言分布极度不均衡（某种语言可能只占全网的千万分之一），加上训练与真实网页的领域差距、语言相似性混淆、以及模型表达能力不足，导致大量噪声文本被错误标记为目标语言。为此，研究团队提出两类补救方案：一是基于词表的可调精度过滤器（发布了约500种语言的精选词表），二是基于Transformer的半监督LangID模型，两者结合后中位数数据集精确率从5.5%跃升至71.2%。这项工作不仅为500多种语言各自构建了10万句以上的相对干净语料，也揭示了现有多语言网络语料库质量被严重高估的问题，为迈向覆盖1000种语言的网络文本语料库奠定了基础。
原文 arXiv:2010.14571;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.14571v2