Is Reinforcement Learning (Not) for Natural Language Processing: Benchmarks, Baselines, and Building Blocks for Natural Language Policy Optimization
Rajkumar Ramamurthy*♠*♠{}^{\textbf{*}\spadesuit} Prithviraj Ammanabrolu*♡*♡{}^{\textbf{*}\heartsuit} Kianté Brantley♣ Jack Hessel♡ Rafet Sifa♠ Christian Bauckhage♠ Hannaneh Hajishirzi♢♡ Yejin Choi♢♡ ♠Fraunhofer IAIS ♡Allen Institute for Artificial Intelligence ♣Cornell University ♢Paul G. Allen School of Computer Science, University of Washington
Abstract
We tackle the problem of aligning pre-trained large language models (LMs) with human preferences. If we view text generation as a sequential decision-making problem, reinforcement learning (RL) appears to be a natural conceptual framework. However, using RL for LM-based generation faces empirical challenges, including training instability due to the combinatorial action space, as well as a lack of open-source libraries and benchmarks customized for LM alignment. Thus, a question rises in the research community: is RL a practical paradigm for NLP?
中文速览
让大语言模型真正符合人类偏好是当前NLP领域的核心挑战，强化学习（Reinforcement Learning, RL）理论上是解决这一问题的自然框架，但因为词汇表动辄数万词导致的巨大动作空间，实际训练极不稳定，加之缺乏统一的开源工具和评测基准，RL在NLP中的可行性一直存疑。为此，研究团队推出了三件套：开源库RL4LMs（支持任意HuggingFace生成模型与多种RL算法）、首个面向RL算法的语言生成评测基准GRUE（用奖励函数而非固定目标字符串来衡量六项生成任务），以及一个名为NLPO（Natural Language Policy Optimization）的新算法——它通过动态学习在每个生成步骤屏蔽低相关词元，将有效动作空间大幅压缩。实验表明，NLPO在自动评测和人工评测上均比PPO等已有策略梯度方法更稳定、更优越，RL训练方式在数据效率和参数效率上也全面优于纯监督学习。这项工作的意义在于：它用一套完整的工具链证明了RL在语言模型对齐上切实可行，为后续让模型输出真正贴近人类偏好的研究提供了坚实的基础设施。
原文 arXiv:2210.01241;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.01241v3