RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encoder
Shitao Xiao1 , Zheng Liu, Yingxia Shao1, Zhao Cao2 1: Beijing University of Posts and Telecommunications, Beijing, China 2: Huawei Technologies Ltd. Co., Shenzhen, China The two researchers make equal contributions to this work and are designated as co-first authors.
Abstract
Despite pre-training’s progress in many important NLP tasks, it remains to explore effective pre-training strategies for dense retrieval. In this paper, we propose RetroMAE, a new retrieval oriented pre-training paradigm based on Masked Auto-Encoder (MAE). RetroMAE is highlighted by three critical designs. 1) A novel MAE workflow, where the input sentence is polluted for encoder and decoder with different masks. The sentence embedding is generated from the encoder’s masked input; then, the original sentence is recovered based on the sentence embedding and the decoder’s masked input via masked language modeling. 2) Asymmetric model structure, with a full-scale BERT like transformer as encoder, and a one-layer transformer as decoder. 3) Asymmetric masking ratios, with a moderate ratio for encoder: 15 $\sim$ 30%, and an aggressive ratio for decoder: 50 $\sim$ 70%. Our framework is simple to realize and empirically competitive: the pre-trained models dramatically improve the SOTA performances on a wide range of dense retrieval benchmarks, like BEIR and MS MARCO. The source code and pre-trained models are made publicly available at https://github.com/staoxiao/RetroMAE so as to inspire m
中文速览
密集检索（dense retrieval）模型通常基于BERT等通用语言模型，但这些模型的预训练任务（如MLM）主要面向词级别，句子级别的语义表示能力不足，导致在检索任务上效果有限。为此，研究者提出了RetroMAE，一种专门面向检索的掩码自编码（Masked Auto-Encoder）预训练框架：编码器用适度掩码率（15–30%）处理输入句子并生成句向量，解码器则以高度激进的掩码率（50–70%）结合该句向量重建原始句子，整体采用"重量级编码器+单层解码器"的非对称结构，迫使编码器学到更丰富的语义。实验表明，RetroMAE在零样本评测集BEIR上平均得分达45.2，在有监督的MS MARCO段落检索任务上MRR@10达41.6，均大幅超越同等规模模型的已有最优结果。这项工作为如何设计检索专用的预训练范式提供了清晰思路，也为后续稠密检索研究提供了开源的强力基线。
原文 arXiv:2205.12035;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.12035v2