Language-agnostic BERT Sentence Embedding
Fangxiaoyu Feng, Yinfei Yang††, Daniel Cer, Naveen Arivazhagan, Wei Wang††Google AI Mountain View {fangxiaoyu, cer, {yangyin7, Equal contributions. Work done while at Google.
Abstract
While BERT is an effective method for learning monolingual sentence embeddings for semantic similarity and embedding based transfer learning Reimers and Gurevych (2019), BERT based cross-lingual sentence embeddings have yet to be explored. We systematically investigate methods for learning multilingual sentence embeddings by combining the best methods for learning monolingual and cross-lingual representations including: masked language modeling (MLM), translation language modeling (TLM) Conneau and Lample (2019), dual encoder translation ranking Guo et al. (2018), and additive margin softmax Yang et al. (2019a). We show that introducing a pre-trained multilingual language model dramatically reduces the amount of parallel training data required to achieve good performance by 80%. Composing the best of these methods produces a model that achieves 83.7% bi-text retrieval accuracy over 112 languages on Tatoeba, well above the 65.5% achieved by Artetxe and Schwenk (2019b), while still performing competitively on monolingual transfer learning benchmarks Conneau and Kiela (2018). Parallel data mined from CommonCrawl using our best model is shown to train competitive NMT models for en-zh a
中文速览
跨语言句子嵌入（cross-lingual sentence embeddings）领域一直缺乏像BERT那样强大的预训练基础，现有模型要么从头训练、要么只覆盖有限语言。研究者将遮蔽语言模型（MLM）和翻译语言模型（TLM）预训练与双编码器（dual encoder）翻译排序微调系统地结合起来，并引入加性边距softmax和跨加速器负采样等技巧，打造出覆盖109+种语言的多语言句子嵌入模型LaBSE。实验表明，引入预训练大模型可将所需平行训练数据减少80%，模型在涵盖112种语言的Tatoeba检索任务上达到83.7%的准确率，大幅超越此前最优的65.5%，同时在单语迁移学习基准上也保持竞争力。这项工作的意义在于，它为低资源语言提供了实用的跨语言语义表示工具，公开发布的模型可直接用于多语言检索、平行语料挖掘和下游分类等场景。
原文 arXiv:2007.01852;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.01852v2