Human-centric dialog training via offline reinforcement learning
Natasha Jaques*12, Judy Hanwen Shen*1, Asma Ghandeharioun1, Craig Ferguson1, Agata Lapedriza1, Noah Jones1, Shixiang Shane Gu2, Rosalind Picard1 *Equal contribution 1Massachusetts Institute of Technology, Cambridge, USA <<judyshen, asma_gh, agata, ncjones, 2Google Research, Mountain View, USA <<natashajaques,
Abstract
How can we train a dialog model to produce better conversations by learning from human feedback, without the risk of humans teaching it harmful chat behaviors? We start by hosting models online, and gather human feedback from real-time, open-ended conversations, which we then use to train and improve the models using offline reinforcement learning (RL). We identify implicit conversational cues including language similarity, elicitation of laughter, sentiment, and more, which indicate positive human feedback, and embed these in multiple reward functions. A well-known challenge is that learning an RL policy in an offline setting usually fails due to the lack of ability to explore and the tendency to make over-optimistic estimates of future reward. These problems become even harder when using RL for language models, which can easily have a 20,000 action vocabulary and many possible reward functions. We solve the challenge by developing a novel class of offline RL algorithms. These algorithms use KL-control to penalize divergence from a pre-trained prior language model, and use a new strategy to make the algorithm pessimistic, instead of optimistic, in the face of uncertainty. We test
中文速览
为了让对话AI从真实人类对话中学习如何聊得更好，同时又不至于被用户"教坏"，研究者搭建了一个在线平台收集真实对话数据，并从中提取语言风格相似度、引发笑声、情感倾向等隐式反馈信号作为奖励，再通过离线强化学习（offline reinforcement learning）来优化对话模型。核心难点在于：离线RL无法与环境实时互动，模型容易对训练数据之外的动作产生过度乐观的错误估计，在词表多达两万词的语言任务中这一问题尤为严峻。为此，研究者提出了一种名为"Way Off-Policy（WOP）"的新算法，用KL散度约束让模型不偏离预训练语言模型的分布，同时借助dropout的多次随机前向传播取最小值，使模型在不确定时保持悲观而非乐观，从而抑制外推误差。在80名真实用户的开放域对话测试中，WOP显著优于现有最强离线深度RL基线，证明这一方法可以安全、有效地利用静态人类反馈数据来提升任何生成式对话模型。
原文 arXiv:2010.05848;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.05848v1