Towards Best Practices for Training Multilingual Dense Retrieval Models
Xinyu Zhang∗, Kelechi Ogueji∗, Xueguang Ma, Jimmy Lin David R. Cheriton School of Computer Science University of Waterloo, Ontario Canada
Abstract
Dense retrieval models using a transformer-based bi-encoder design have emerged as an active area of research. In this work, we focus on the task of monolingual retrieval in a variety of typologically diverse languages using one such design. Although recent work with multilingual transformers demonstrates that they exhibit strong cross-lingual generalization capabilities, there remain many open research questions, which we tackle here. Our study is organized as a “best practices” guide for training multilingual dense retrieval models, broken down into three main scenarios: where a multilingual transformer is available, but relevance judgments are not available in the language of interest; where both models and training data are available; and, where training data are available not but models. In considering these scenarios, we gain a better understanding of the role of multi-stage fine-tuning, the strength of cross-lingual transfer under various conditions, the usefulness of out-of-language data, and the advantages of multilingual vs. monolingual transformers. Our recommendations offer a guide for practitioners building search applications, particularly for low-resource languages,
中文速览
多语言稠密检索（dense retrieval）模型在低资源语言上的训练一直缺乏系统性指导，研究者不清楚在没有目标语言标注数据、或兼有数据与模型、或只有数据没有预训练模型等不同情况下该怎么做。本文以多语言BERT（mBERT）为骨干构建双编码器检索模型（mDPR），在覆盖11种类型差异显著语言的Mr. TYDI基准上，系统探索了多阶段微调、跨语言迁移能力边界、域外语言数据的价值以及多语言与单语言Transformer的优劣。研究发现，先用英文MS MARCO做"预微调"再在目标语言上继续训练几乎总是有益或无害，即便目标语言与英语在语系和文字上毫无关联；跨语言迁移的有效性与不同语言语料间的词片段重叠程度密切相关，这也解释了一些反直觉的现象，比如在泰语上微调的mBERT对阿拉伯语检索仍然有效。这些发现为需要为低资源语言快速搭建搜索应用的从业者提供了一套有实证支撑的"最佳实践"指南，也为后续多语言检索研究奠定了坚实基础。
原文 arXiv:2204.02363;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.02363v1