SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient
Lantao Yu†, Weinan Zhang†, Jun Wang‡, Yong Yu† †Shanghai Jiao Tong University, ‡University College London Weinan Zhang is the corresponding author.
Abstract
As a new way of training generative models, Generative Adversarial Net (GAN) that uses a discriminative model to guide the training of the generative model has enjoyed considerable success in generating real-valued data. However, it has limitations when the goal is for generating sequences of discrete tokens. A major reason lies in that the discrete outputs from the generative model make it difficult to pass the gradient update from the discriminative model to the generative model. Also, the discriminative model can only assess a complete sequence, while for a partially generated sequence, it is non-trivial to balance its current score and the future one once the entire sequence has been generated. In this paper, we propose a sequence generation framework, called SeqGAN, to solve the problems. Modeling the data generator as a stochastic policy in reinforcement learning (RL), SeqGAN bypasses the generator differentiation problem by directly performing gradient policy update. The RL reward signal comes from the GAN discriminator judged on a complete sequence, and is passed back to the intermediate state-action steps using Monte Carlo search. Extensive experiments on synthetic data an
中文速览
把序列生成（如文本、诗歌、音乐）问题塞进GAN框架时，有两道坎：生成器输出的是离散token，梯度根本没法从判别器反向传回去；而且判别器只能给完整序列打分，生成到一半的序列无从评估。SeqGAN把生成器重新理解为强化学习中的随机策略，每一步选词就是一次"动作"，并用蒙特卡洛搜索（Monte Carlo search）把只有完整序列才有的判别器奖励信号回传到每个中间步骤，从而绕开了离散数据不可微的死结。在合成数据和诗歌生成、对话、音乐创作三项真实任务上，SeqGAN都大幅超越了最大似然、Scheduled Sampling等强基线，甚至在人类专家评分上也有显著优势。这项工作第一次打通了GAN与离散序列生成之间的通路，为后续文本生成领域的对抗训练研究奠定了基础。
原文 arXiv:1609.05473;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1609.05473v6