Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models
Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B. Hall, Daniel Cer, Yinfei Yang Google Research Mountain View, CA
Abstract
We provide the first exploration of sentence embeddings from text-to-text transformers (T5). Sentence embeddings are broadly useful for language processing tasks. While T5 achieves impressive performance on language tasks cast as sequence-to-sequence mapping problems, it is unclear how to produce sentence embeddings from encoder-decoder models. We investigate three methods for extracting T5 sentence embeddings: two utilize only the T5 encoder and one uses the full T5 encoder-decoder model. To support our investigation, we establish a new sentence representation transfer benchmark, SentGLUE, which extends the SentEval toolkit to nine tasks from the GLUE benchmark (Wang et al., 2018). Our encoder-only models outperforms Sentence-BERT (Reimers and Gurevych, 2019) and SimCSE (Gao et al., 2021) sentence embeddings on both SentEval and SentGLUE transfer tasks, including semantic textual similarity (STS). Scaling up T5 from millions to billions of parameters is found to produce consistent further improvements. Finally, our encoder-decoder method achieves a new state-of-the-art on STS when using sentence embeddings.111Our models are released at https://tfhub.dev/google/collections/sentence
中文速览
让大型文本生成模型T5也能产出高质量句子嵌入（sentence embeddings），是一个此前从未被系统探索过的问题，因为T5采用编码器-解码器结构，与常见的仅编码器模型（如BERT）不同，不能直接套用现有方法。研究团队提出了三种从T5中提取句子嵌入的策略——仅取编码器首个词元、对编码器全部词元做均值池化、以及取解码器首个词元——并结合对比学习和两阶段微调（先在问答数据上训练，再在自然语言推理数据上精调）来提升表示质量。实验在SentEval及新建的SentGLUE基准上表明，即便不经微调，T5编码器的均值池化嵌入就已超越经过微调的SimCSE-RoBERTa；经过微调后，编码器-解码器方案在语义文本相似度（STS）任务上刷新了当时的最优记录，且将模型参数从数亿扩展至110亿时性能持续稳步提升。这项工作首次证明了超大规模文本生成预训练模型可以直接用于高效的句子表示学习，为检索、聚类等对嵌入质量敏感的任务提供了更强大的基础模型选项。
原文 arXiv:2108.08877;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2108.08877v3