Hierarchical Reinforcement Learning for Open-Domain Dialog
Abdelrhman Saleh,1, Natasha Jaques,2,* Asma Ghandeharioun,2 Judy Hanwen Shen,2 Rosalind Picard2 1Harvard University, 2MIT Media Lab Equal Contribution
Abstract
Open-domain dialog generation is a challenging problem; maximum likelihood training can lead to repetitive outputs, models have difficulty tracking long-term conversational goals, and training on standard movie or online datasets may lead to the generation of inappropriate, biased, or offensive text. Reinforcement Learning (RL) is a powerful framework that could potentially address these issues, for example by allowing a dialog model to optimize for reducing toxicity and repetitiveness. However, previous approaches which apply RL to open-domain dialog generation do so at the word level, making it difficult for the model to learn proper credit assignment for long-term conversational rewards. In this paper, we propose a novel approach to hierarchical reinforcement learning (HRL), VHRL, which uses policy gradients to tune the utterance-level embedding of a variational sequence model. This hierarchical approach provides greater flexibility for learning long-term, conversational rewards. We use self-play and RL to optimize for a set of human-centered conversation metrics, and show that our approach provides significant improvements – in terms of both human evaluation and automatic metri
中文速览
对话系统长期以来面临三大顽疾：最大似然训练产生枯燥重复的回复、模型难以追踪对话的长期目标、以及在网络或影视台词数据上训练容易生成有害或偏激内容。为解决这些问题，研究者提出了一种名为 VHRL（变分层次强化学习，Variational Hierarchical Reinforcement Learning）的新方法：在一个层次化变分序列模型（VHRED）的基础上，不再像以往那样对每个词施加强化学习（Reinforcement Learning）信号，而是用策略梯度直接调整句子级别的潜变量分布，让模型在更高的抽象层次上学习长期对话奖励。具体训练时，模型通过自我对话（self-play）生成完整对话，再以毒性（toxicity）、重复度、情感积极性、对用户输入的回应度等以人为中心的指标作为奖励信号来优化模型。在自动评测和真实用户在线交互评测两个维度上，VHRL 均显著优于包括 Transformer 在内的当前最优对话模型，表明层次化强化学习是同时提升对话安全性与质量的有效路径。
原文 arXiv:1909.07547;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.07547v3