Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation
Nils Reimers and Iryna Gurevych Ubiquitous Knowledge Processing Lab (UKP-TUDA) Department of Computer Science, Technische Universität Darmstadt www.ukp.tu-darmstadt.de
Abstract
We present an easy and efficient method to extend existing sentence embedding models to new languages. This allows to create multilingual versions from previously monolingual models. The training is based on the idea that a translated sentence should be mapped to the same location in the vector space as the original sentence. We use the original (monolingual) model to generate sentence embeddings for the source language and then train a new system on translated sentences to mimic the original model. Compared to other methods for training multilingual sentence embeddings, this approach has several advantages: It is easy to extend existing models with relatively few samples to new languages, it is easier to ensure desired properties for the vector space, and the hardware requirements for training are lower. We demonstrate the effectiveness of our approach for 50+ languages from various language families. Code to extend sentence embeddings models to more than 400 languages is publicly available.111Code, models, and datasets: https://github.com/UKPLab/sentence-transformers
中文速览
让多语言句子嵌入（sentence embedding）模型的训练又快又省资源，是NLP领域长期面临的难题。这项工作提出了一种叫做"多语言知识蒸馏（multilingual knowledge distillation）"的方法：先用一个训练好的英文句子嵌入模型（教师模型）为源语言句子生成向量，再让一个多语言学生模型去模仿这些向量，同时也让翻译后的目标语言句子映射到同一位置——简单说，就是让翻译对等的句子在向量空间里"住到一起"。实验覆盖50多种语言，在跨语言语义相似度、平行句检索等任务上全面超越LASER、mUSE、LaBSE等主流方案，对低资源语言的提升尤为显著，最高达40个准确率百分点。这项工作的意义在于：只需少量平行语料和较低的硬件门槛，就能把任意现有的单语句子嵌入模型快速扩展到400多种语言，代码与模型已公开发布，极大降低了多语言NLP应用的开发成本。
原文 arXiv:2004.09813;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.09813v2