Multi-view and Multi-source Transfers in Neural Topic Modeling with Pretrained Topic and Word Embeddings
Pankaj Gupta1,2, Yatin Chaudhary1, Hinrich Schütze2 1Corporate Technology, Machine-Intelligence (MIC-DE), Siemens AG Munich, Germany 2CIS, University of Munich (LMU) Munich, Germany {pankaj.gupta,
Abstract
Though word embeddings and topics are complementary representations, several past works have only used pre-trained word embeddings in (neural) topic modeling to address data sparsity problem in short text or small collection of documents. However, no prior work has employed (pre-trained latent) topics in transfer learning paradigm. In this paper, we propose an approach to (1) perform knowledge transfer using latent topics obtained from a large source corpus, and (2) jointly transfer knowledge via the two representations (or views) in neural topic modeling to improve topic quality, better deal with polysemy and data sparsity issues in a target corpus. In doing so, we first accumulate topics and word representations from one or many source corpora to build a pool of topics and word vectors. Then, we identify one or multiple relevant source domain(s) and take advantage of corresponding topics and word features via the respective pools to guide meaningful learning in the sparse target domain. We quantify the quality of topic and document representations via generalization (perplexity), interpretability (topic coherence) and information retrieval (IR) using short-text, long-text, small
中文速览
针对稀疏文本（短文本或小规模语料）中主题模型效果差的问题，研究者提出了一种多视角、多源迁移学习框架，将来自大规模源语料的两类知识——词嵌入（局部视角，捕捉局部语义）和潜在主题特征（全局视角，捕捉文档级主题结构）——联合迁移到目标域的神经主题模型（DocNADE）中，同时通过主题对齐机制从多个相关源域中筛选有益知识、抑制负迁移。在新闻和医疗领域的7个目标语料（含短文本、长文本、大小规模数据集）上，该框架在困惑度、主题连贯性和信息检索三项指标上均达到当时最优水平。这项工作的意义在于，它首次将"主题迁移"引入无监督主题建模，弥补了以往方法仅依赖词嵌入、忽视主题级别全局语义的缺陷，为低资源场景下的文本表示学习提供了更完整的解决思路。
原文 arXiv:1909.06563;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.06563v2