Contextual LSTM (CLSTM) models for Large scale NLP tasks
Shalini Ghosh Oriol Vinyals Brian Strope Scott Roy Tom Dean Larry Heck
Abstract
Documents exhibit sequential structure at multiple levels of abstraction (e.g., sentences, paragraphs, sections). These abstractions constitute a natural hierarchy for representing the context in which to infer the meaning of words and larger fragments of text. In this paper, we present CLSTM (Contextual LSTM), an extension of the recurrent neural network LSTM (Long-Short Term Memory) model, where we incorporate contextual features (e.g., topics) into the model. We evaluate CLSTM on three specific NLP tasks: word prediction, next sentence selection, and sentence topic prediction. Results from experiments run on two corpora, English documents in Wikipedia and a subset of articles from a recent snapshot of English Google News, indicate that using both words and topics as features improves performance of the CLSTM models over baseline LSTM models for these tasks. For example on the next sentence selection task, we get relative accuracy improvements of 21% for the Wikipedia dataset and 18% for the Google News dataset. This clearly demonstrates the significant benefit of using context appropriately in natural language (NL) tasks. This has implications for a wide variety of NL applicatio
中文速览
语言模型在预测词语时往往只盯着当前句子里的词序列，却忽略了文章更宏观的主题背景，导致在需要长程上下文理解的场景中表现不佳。为此，研究者在经典的 LSTM（长短期记忆网络）基础上引入主题向量，将句子或段落级别的话题信息直接拼接进每个 LSTM 单元的输入，构建出名为 CLSTM（Contextual LSTM，上下文感知长短期记忆网络）的新模型。在维基百科和谷歌新闻两个大规模英文语料上，针对词预测、下一句选择和句子主题预测三项任务进行评测，CLSTM 均优于不含主题特征的基线 LSTM，其中下一句选择任务的相对准确率分别提升了 21%（维基百科）和 18%（谷歌新闻）。这项工作表明，将层次化的长程上下文信号融入序列模型，能为问答、句子补全、对话系统等众多自然语言处理应用带来显著收益。
原文 arXiv:1602.06291;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1602.06291v2