Language as a Latent Variable: Discrete Generative Models for Sentence Compression

中文速览

句子压缩既要缩短原句，又要保留关键信息，但只靠成对标注的例句训练，往往很难充分利用大量没有摘要标注的文本。研究者让模型先从原句中挑出一个简短句子作为“潜在摘要”，再用它重建原句，并结合指针网络、强化学习和有监督训练，让模型学会挑词、写出通顺的压缩句。实验显示，标注数据充足时，这种方法在抽取式和改写式压缩上都取得领先结果；标注数据较少时，加入未标注文本也能达到与以往监督模型相近的水平。它的重要性在于，模型不只会从标注例句中学习，还能借助普通文本，减少对昂贵人工标注的依赖。

摘要

In this work we explore deep generative models of text in which the latent representation of a document is itself drawn from a discrete language model distribution. We formulate a variational auto-encoder for inference in this model and apply it to the task of compressing sentences. In this application the generative model first draws a latent summary sentence from a background language model, and then subsequently draws the observed sentence conditioned on this latent summary. In our empirical evaluation we show that generative formulations of both abstractive and extractive compression yield state-of-the-art results when trained on a large amount of supervised data. Further, we explore semi-supervised compression scenarios where we show that it is possible to achieve performance competitive with previously proposed supervised models while training on a fraction of the supervised data.

术语表

Deep Generative Model

深度生成模型

Discrete Language Model

离散语言模型

Variational Auto-Encoder (VAE)

变分自编码器（VAE）

Sentence Compression

句子压缩

Abstractive Compression

抽象式压缩

Extractive Compression

抽取式压缩

Semi-Supervised Learning

半监督学习

Sequence-to-Sequence

序列到序列

Attention

注意力

Joint Distribution

联合分布

Auto-Encoder

自编码器

Generative Auto-Encoding Sentence Compression (ASC)

生成式自编码句子压缩（ASC）

Latent Language Model

潜在语言模型

Posterior Distribution

后验分布

Inference Network

推断网络

Variational Lower Bound

变分下界

Reparameterisation Trick

重参数化技巧

REINFORCE

REINFORCE

Pointer Network

指针网络

Forced-Attention Sentence Compression (FSC)

强制注意力句子压缩（FSC）

Gigaword

Gigaword

Bidirectional LSTM

双向长短期记忆网络

Soft Attention

软注意力

Language Model Prior

语言模型先验

Variational Inference

变分推断