Learning General Purpose Distributed Sentence Representations via Large Scale Multi-task Learning
Sandeep Subramanian1,2,3, Adam Trischler3, Yoshua Bengio1,2,4、Christopher J Pal1,5 1 Montréal Institute for Learning Algorithms (MILA) 2 Université de Montréal 3 Microsoft Research Montreal 4 CIFAR Senior Fellow 5 École Polytechnique de Montréal Work done while author was an intern at Microsoft Research Montreal
Abstract
A lot of the recent success in natural language processing (NLP) has been driven by distributed vector representations of words trained on large amounts of text in an unsupervised manner. These representations are typically used as general purpose features for words across a range of NLP problems. However, extending this success to learning representations of sequences of words, such as sentences, remains an open problem. Recent work has explored unsupervised as well as supervised learning techniques with different training objectives to learn general purpose fixed-length sentence representations. In this work, we present a simple, effective multi-task learning framework for sentence representations that combines the inductive biases of diverse training objectives in a single model. We train this model on several data sources with multiple training objectives on over 100 million sentences. Extensive experiments demonstrate that sharing a single recurrent sentence encoder across weakly related tasks leads to consistent improvements over previous methods. We present substantial improvements in the context of transfer learning and low-resource settings using our learned general-purpos
中文速览
用一个共享的循环神经网络同时学习多种任务，能不能得到更通用的句子表示？研究者提出了一个"一对多"的多任务学习框架（GenSen），让同一个双向GRU编码器在机器翻译、自然语言推理、成分句法分析、跳跃思维预测等六种任务上联合训练，用超过1亿句话同时优化多个目标，从而把不同任务各自的归纳偏置（inductive bias）都融入同一个定长句子向量。在迁移学习基准测试上，GenSen持续超越此前的单任务方法；在低资源场景下表现尤为突出——在Quora重复问题数据集上，仅用6%的标注数据就能达到从头训练全量数据的相当水平。这项工作表明，把足够多样的训练信号汇聚到单一编码器中，是获得高质量通用句子表示的有效途径，对数据稀缺场景下的NLP应用具有重要的实用价值。
原文 arXiv:1804.00079;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1804.00079v1