Decoding Methods for Neural Narrative Generation
Alexandra DeLucia ∗Equal contribution. Aaron Mueller∗ Johns Hopkins University {aadelucia,、Xiang Lisa Li Stanford University \ANDJoão Sedoc† New York University †Work performed while at Johns Hopkins University.
Abstract
Narrative generation is an open-ended NLP task in which a model generates a story given a prompt. The task is similar to neural response generation for chatbots; however, innovations in response generation are often not applied to narrative generation, despite the similarity between these tasks. We aim to bridge this gap by applying and evaluating advances in decoding methods for neural response generation to neural narrative generation. In particular, we employ GPT-2 and perform ablations across nucleus sampling thresholds and diverse decoding hyperparameters—specifically, maximum mutual information—analyzing results over multiple criteria with automatic and human evaluation. We find that (1) nucleus sampling is generally best with thresholds between 0.7 and 0.9; (2) a maximum mutual information objective can improve the quality of generated stories; and (3) established automatic metrics do not correlate well with human judgments of narrative quality on any qualitative metric.
中文速览
故事自动生成（narrative generation）和聊天机器人回复生成在任务结构上高度相似，但后者发展出的许多优秀解码技术从未被移植到前者。本文用 GPT-2 在 WritingPrompts 数据集上进行微调，系统地测试了核采样（nucleus sampling）的不同概率阈值，以及借鉴自对话生成的最大互信息（Maximum Mutual Information, MMI）多样化解码目标对长篇故事质量的影响，并同时采用自动指标和人工评估来衡量结果。研究发现：核采样阈值在 0.7 到 0.9 之间效果最佳；引入 MMI 目标能在一定程度上提升生成故事的质量；而现有自动评估指标（如困惑度、词汇多样性、句子相似度）与人类对故事趣味性、连贯性等维度的判断几乎没有相关性。这最后一点尤其值得重视——它说明当前用于衡量生成质量的自动化工具并不适用于长篇创意写作场景，亟需针对叙事生成开发更可靠的评估方法。
原文 arXiv:2010.07375;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.07375v2