An Actor-Critic Algorithm for Sequence Prediction
Dzmitry Bahdanau Philemon Brakel Kelvin Xu Anirudh Goyal Université de Montréal、Ryan Lowe Joelle Pineau McGill University \ANDAaron Courville Université de Montréal、Yoshua Bengio Université de Montréal CIFAR Senior FellowCIFAR Fellow
Abstract
We present an approach to training neural networks to generate sequences using actor-critic methods from reinforcement learning (RL). Current log-likelihood training methods are limited by the discrepancy between their training and testing modes, as models must generate tokens conditioned on their previous guesses rather than the ground-truth tokens. We address this problem by introducing a critic network that is trained to predict the value of an output token, given the policy of an actor network. This results in a training procedure that is much closer to the test phase, and allows us to directly optimize for a task-specific score such as BLEU. Crucially, since we leverage these techniques in the supervised learning setting rather than the traditional RL setting, we condition the critic network on the ground-truth output. We show that our method leads to improved performance on both a synthetic task, and for German-English machine translation. Our analysis paves the way for such methods to be applied in natural language generation tasks, such as machine translation, caption generation, and dialogue modelling.
中文速览
用神经网络生成序列时，传统的"教师强制"训练方式让模型每步都看着正确答案学，但测试时却要靠自己之前的输出来继续生成，这种训练和测试之间的脱节会导致错误不断积累、性能下降。为了解决这一问题，作者借鉴强化学习中的"演员-评论家"（actor-critic）框架，在原有的序列生成网络（演员）之外额外训练一个评论家网络，让它预测每个生成token在未来能带来多高的任务得分（如BLEU），再用这个预测值来直接引导演员改进输出。与纯强化学习不同的是，由于训练时存在标准答案，评论家网络可以直接看到正确输出，从而提供更准确的价值估计，并通过时序差分（temporal difference）方法降低学习方差。实验表明，该方法在合成任务和德英机器翻译任务上均优于最大似然训练和REINFORCE基线，为序列生成领域更广泛地应用强化学习技术提供了可行路径。
原文 arXiv:1607.07086;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1607.07086v3