Portuguese Word Embeddings: Evaluating on Word Analogies and Natural Language Tasks
Nathan S. Hartmann1, Erick Fonseca1, Christopher D. Shulby1, Marcos V. Treviso1, Jéssica S. Rodrigues2, Sandra M. Aluísio1
Abstract
Word embeddings have been found to provide meaningful representations for words in an efficient way; therefore, they have become common in Natural Language Processing systems. In this paper, we evaluated different word embedding models trained on a large Portuguese corpus, including both Brazilian and European variants. We trained 31 word embedding models using FastText, GloVe, Wang2Vec and Word2Vec. We evaluated them intrinsically on syntactic and semantic analogies and extrinsically on POS tagging and sentence semantic similarity tasks. The obtained results suggest that word analogies are not appropriate for word embedding evaluation; task-specific evaluations appear to be a better option.
中文速览
针对葡萄牙语词向量（word embeddings）缺乏系统性评估的问题，研究者收集了涵盖巴西葡语和欧洲葡语的大规模语料库，分别用 FastText、GloVe、Wang2Vec 和 Word2Vec 四种算法训练了 31 个不同维度的词向量模型，再从内部评估（句法与语义类比任务）和外部评估（词性标注、句子语义相似度两项实际 NLP 任务）两个角度对这些模型进行横向比较。结果显示，在类比任务上表现最好的模型并不一定在下游任务中同样出色——例如语义类比最差的 CBOW 模型却在句子相似度任务中拔得头筹，而在形态特征上有天然优势的 FastText 在词性标注中的表现反而令人失望。这一发现印证了"词类比测试不适合作为词向量评估标准"的观点，表明任务导向的外部评估才是衡量词向量质量的更可靠方式；与此同时，研究者公开了全部 31 个葡萄牙语词向量模型，为后续研究提供了重要的基础资源。
原文 arXiv:1708.06025;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1708.06025v1