Massively Multilingual Document Alignment with Cross-lingual Sentence-Mover’s Distance
Ahmed El-Kishky Facebook AI、Francisco Guzmán Facebook AI
Abstract
Document alignment aims to identify pairs of documents in two distinct languages that are of comparable content or translations of each other. Such aligned data can be used for a variety of NLP tasks from training cross-lingual representations to mining parallel data for machine translation. In this paper we develop an unsupervised scoring function that leverages cross-lingual sentence embeddings to compute the semantic distance between documents in different languages. These semantic distances are then used to guide a document alignment algorithm to properly pair cross-lingual web documents across a variety of low, mid, and high-resource language pairs. Recognizing that our proposed scoring function and other state of the art methods are computationally intractable for long web documents, we utilize a more tractable greedy algorithm that performs comparably. We experimentally demonstrate that our distance metric performs better alignment than current baselines outperforming them by 7% on high-resource language pairs, 15% on mid-resource language pairs, and 22% on low-resource language pairs.
中文速览
两种语言的网页文档如何自动配对——即跨语言文档对齐——是获取平行语料的关键步骤，而现有方法要么依赖元数据和文档结构等不可靠特征，要么只在英法这类高资源语言对上有效。研究者提出了一种名为"跨语言句子移动距离"（Cross-Lingual Sentence Mover's Distance，SMD）的无监督评分方法：先用多语言句子编码器 LASER 将文档中每个句子编码到同一语义空间，再借鉴"推土机距离"（Earth Mover's Distance）的思路，计算把源文档的句子分布"搬运"成目标文档句子分布所需的最小代价，以此衡量两篇跨语言文档的语义相似度；同时引入句子长度加权和逆文档频率加权来降低导航栏、标题等低信息量文本的干扰，并设计了贪心近似算法解决原始算法在长文档上计算量过大的问题。实验结果显示，该方法在高、中、低资源语言对上分别比现有最优基线提升了 7%、15% 和 22%，尤其在低资源场景下优势显著，为机器翻译等下游任务提供了更高质量的跨语言平行数据挖掘手段。
原文 arXiv:2002.00761;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2002.00761v2