Rethinking Skip-thought: A Neighborhood based Approach
Shuai Tang1, Hailin Jin2, Chen Fang2, Zhaowen Wang2, Virginia R. de Sa1 1Department of Cognitive Science, UC San Diego, La Jolla CA 92093, USA 2Adobe Research, 345 Park Ave., San Jose CA 95110, USA
Abstract
We study the skip-thought model proposed by Kiros et al. (2015) with neighborhood information as weak supervision. More specifically, we propose a skip-thought neighbor model to consider the adjacent sentences as a neighborhood. We train our skip-thought neighbor model on a large corpus with continuous sentences, and then evaluate the trained model on 7 tasks, which include semantic relatedness, paraphrase detection, and classification benchmarks. Both quantitative comparison and qualitative investigation are conducted. We empirically show that, our skip-thought neighbor model performs as well as the skip-thought model on evaluation tasks. In addition, we found that, incorporating an autoencoder path in our model didn’t aid our model to perform better, while it hurts the performance of the skip-thought model.
中文速览
句子表示学习（sentence representation learning）领域长期依赖skip-thought模型——用一个编码器压缩当前句子，再用两个独立解码器分别重建前后两句——但这套双解码器设计其实隐含着一个未经验证的假设：前一句和后一句必须从不同的条件分布中生成。研究者提出"skip-thought neighbor模型"，把两个解码器合并为一个，让它同时重建前后邻居句子，本质上是假设相邻句子提供的是同质的邻域信息。在包含7400万句子的BookCorpus上训练后，新模型在语义相关性、复述检测和多项文本分类共7个下游任务上的表现与原始skip-thought模型持平，同时还发现加入自编码路径（让解码器同时重建输入句本身）对新模型没有帮助、对原始模型反而有损；此外，只预测下一句的简化版本在所有变体中表现最佳。这项工作说明skip-thought中的双解码器设计是冗余的，用更简洁的单解码器架构就能达到同等效果，为后续更高效的无监督句子表示学习提供了有价值的参考。
原文 arXiv:1706.03146;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.03146v1