Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representations
Andreas Rückl醆\dagger, Steffen Eger††\dagger, Maxime Peyrard†‡†‡\dagger\ddagger, Iryna Gurevych†‡†‡\dagger\ddagger ††\daggerUbiquitous Knowledge Processing Lab (UKP) ‡‡\ddaggerResearch Training Group AIPHES Department of Computer Science, Technische Universität Darmstadt ††\dagger www.ukp.tu-darmstadt.de ‡‡\ddagger www.aiphes.tu-darmstadt.de
Abstract
Average word embeddings are a common baseline for more sophisticated sentence embedding techniques. However, they typically fall short of the performances of more complex models such as InferSent. Here, we generalize the concept of average word embeddings to power mean word embeddings. We show that the concatenation of different types of power mean word embeddings considerably closes the gap to state-of-the-art methods monolingually and substantially outperforms these more complex techniques cross-lingually. In addition, our proposed method outperforms different recently proposed baselines such as SIF and Sent2Vec by a solid margin, thus constituting a much harder-to-beat monolingual baseline. Our data and code are publicly available.111https://github.com/UKPLab/arxiv2018-xling-sentence-embeddings
中文速览
把句子表示成向量一直是NLP的基础任务，最简单的做法是把句中所有词向量取算术平均，但效果远不如InferSent这类复杂模型。本文把"平均"这个概念推广为**幂均值（power mean）**，即用不同参数 p 计算词向量序列的广义均值（算术均值、最大值、最小值等），再把多种幂均值以及多套异质词向量（GloVe、Word2Vec、Attract-Repel等）拼接起来，构成"拼接幂均值句向量（concatenated power mean embeddings）"。实验覆盖9个单语迁移任务和多个跨语言任务，结果显示该方法在单语场景下大幅超越SIF、Sent2Vec等强基线，与InferSent的差距也明显缩小，而在跨语言场景下则全面超过所有对比方法。这项工作的价值在于：它几乎不需要额外训练，计算极为简单，却能逼近甚至超越需要大规模标注数据和GPU训练的复杂模型，为研究者提供了一个更难被超越的强基线，同时也是首个系统评估通用跨语言句向量的工作。
原文 arXiv:1803.01400;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.01400v2