WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipedia
Holger Schwenk Facebook AI、Vishrav Chaudhary Facebook AI、Shuo Sun Johns Hopkins University、Hongyu Gong University of Illinois at Urbana-Champaign、Francisco Guzmán Facebook AI
Abstract
We present an approach based on multilingual sentence embeddings to automatically extract parallel sentences from the content of Wikipedia articles in 85 languages, including several dialects or low-resource languages. We do not limit the the extraction process to alignments with English, but systematically consider all possible language pairs. In total, we are able to extract 135M parallel sentences for 1620 different language pairs, out of which only 34M are aligned with English. This corpus of parallel sentences is freely available.111https://github.com/facebookresearch/LASER/tree/master/tasks/WikiMatrix
中文速览
为了给低资源和非英语语言对提供更多平行语料，研究者利用多语言句子嵌入技术（LASER）从覆盖85种语言的整个维基百科中自动挖掘平行句对，并首次系统性地考虑所有语言对之间的对齐，而不仅限于与英语的对齐。他们借助基于"边际准则"（margin criterion）的评分方法筛选高质量翻译对，并用FAISS快速相似度搜索库解决了亿级句子规模带来的计算挑战。最终从1620个语言对中共提取了1.35亿条平行句对，其中仅约3400万条涉及英语，并在此基础上训练了1886个语言对的神经机器翻译系统，在TED测试集上取得了良好的BLEU分数，尤其对于那些以往需要经由英语进行"桥接翻译"的远缘语言对效果突出。这份免费公开的WikiMatrix语料库有望显著推动低资源语言的自然语言处理研究，使更多语言能够绕开英语直接建立高质量的机器翻译系统。
原文 arXiv:1907.05791;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1907.05791v2