DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer
Haozhe Ji1, Minlie Huang1 1Department of Computer Science and Technology, Institute for Artificial Intelligence, State Key Lab of Intelligent Technology and Systems, Beijing National Research Center for Information Science and Technology, Tsinghua University, Beijing 100084, China Corresponding author
Abstract
Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware discrete variational Transformer to tackle the incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes the global structure of the text and then applies it to guide the generation process at each decoding step. To further embed discourse-aware information into the discrete latent representations, we introduce an auxiliary objective to model the discourse relations within the text. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.111The source code is available at https://github.com/cdjhz/DiscoDVT.
中文速览
预训练语言模型在生成几十句话的长篇文本时，常常"跑题"或前后逻辑脱节，根本原因是缺乏对全局篇章结构的把握。为此，研究者提出了 DiscoDVT（话语感知离散变分 Transformer），让模型先从文本中学出一段离散隐变量序列来概括全文的高层结构，再把这些"结构码"注入解码器逐步引导生成；同时引入一个辅助训练目标，把来自宾州话语树库（PDTB）的因果、时序等显式话语关系嵌入这些离散表示中，使其真正捕捉段落间的衔接逻辑。在两个开放式故事生成数据集上的自动评测与人工评测均表明，DiscoDVT 在长程连贯性上显著优于基线，且离散编码与篇章结构之间呈现出可解释的对应关系。这项工作为大规模预训练模型与离散结构化隐变量的结合提供了一条可行路径，对长文本生成领域具有重要参考价值。
原文 arXiv:2110.05999;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.05999v1