Universal Sentence Encoder
Daniel Cera𝑎a, Yinfei Yanga𝑎a, Sheng-yi Konga𝑎a, Nan Huaa𝑎a, Nicole Limtiacob𝑏b, Rhomni St. Johna𝑎a, Noah Constanta𝑎a, Mario Guajardo-Céspedesa𝑎a, Steve Yuanc𝑐c, Chris Tara𝑎a, Yun-Hsuan Sunga𝑎a, Brian Stropea𝑎a, Ray Kurzweila𝑎a \ANDa𝑎aGoogle Research Mountain View, CA \Andb𝑏bGoogle Research New York, NY \Andc𝑐cGoogle Cambridge, MA
Abstract
We present models for encoding sentences into embedding vectors that specifically target transfer learning to other NLP tasks. The models are efficient and result in accurate performance on diverse transfer tasks. Two variants of the encoding models allow for trade-offs between accuracy and compute resources. For both variants, we investigate and report the relationship between model complexity, resource consumption, the availability of transfer task training data, and task performance. Comparisons are made with baselines that use word level transfer learning via pretrained word embeddings as well as baselines do not use any transfer learning. We find that transfer learning using sentence embeddings tends to outperform word level transfer. With transfer learning via sentence embeddings, we observe surprisingly good performance with minimal amounts of supervised training data for a transfer task. We obtain encouraging results on Word Embedding Association Tests (WEAT) targeted at detecting model bias. Our pre-trained sentence encoding models are made freely available for download and on TF Hub.
中文速览
句子嵌入（sentence embedding）领域长期依赖词向量做迁移学习，但句子级别的语义表示往往能带来更强的泛化能力，谷歌研究团队为此提出了"通用句子编码器"（Universal Sentence Encoder），旨在训练出一个可直接迁移到各类自然语言处理下游任务的句子向量模型。他们设计了两种编码器：一种基于Transformer架构，精度更高但计算开销随句长呈平方级增长；另一种基于深度平均网络（Deep Averaging Network, DAN），计算量随句长线性增长、速度更快但精度略低；两者均采用多任务学习，同时在无监督文本和斯坦福自然语言推理语料上训练，最终输出512维向量。实验覆盖情感分类、问题分类、语义相似度等多项任务，结果显示句子级迁移学习全面优于仅用词向量的迁移方式，尤其在标注数据极少时优势最为突出，将句子向量与词向量拼接使用可获得最佳综合性能。这项工作的意义在于提供了开箱即用的预训练句子编码器，让数据匮乏的NLP场景也能以极低成本获得高质量的语义表示，同时对模型偏见问题也给出了初步量化分析。
原文 arXiv:1803.11175;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.11175v2