RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
Harrison Lee Samrat Phatale Hassan Mansoor Thomas Mesnard Johan Ferret Kellie Lu Colton Bishop Ethan Hall Victor Carbune Abhinav Rastogi Sushant Prakash
Abstract
Reinforcement learning from human feedback (RLHF) has proven effective in aligning large language models (LLMs) with human preferences, but gathering high-quality preference labels is expensive. RL from AI Feedback (RLAIF), introduced in Bai et al. (2022b), offers a promising alternative that trains the reward model (RM) on preferences generated by an off-the-shelf LLM. Across the tasks of summarization, helpful dialogue generation, and harmless dialogue generation, we show that RLAIF achieves comparable performance to RLHF. Furthermore, we take a step towards “self-improvement” by demonstrating that RLAIF can outperform a supervised fine-tuned baseline even when the AI labeler is the same size as the policy, or even the exact same checkpoint as the initial policy. Finally, we introduce direct-RLAIF (d-RLAIF) - a technique that circumvents RM training by obtaining rewards directly from an off-the-shelf LLM during RL, which achieves superior performance to canonical RLAIF. Our results suggest that RLAIF can achieve performance on-par with using human feedback, offering a potential solution to the scalability limitations of RLHF.
中文速览
用AI生成的偏好标签来替代昂贵的人工标注，一直是大语言模型对齐领域的关键挑战。这项研究系统比较了"从AI反馈强化学习"（RLAIF）与传统"从人类反馈强化学习"（RLHF）在摘要生成、有益对话和无害对话三项任务上的表现，发现两者训练出的模型被人类评测者选中的概率几乎没有统计差异，说明AI标注可以直接替代人工标注。研究还提出了一种更简洁的"直接RLAIF"（d-RLAIF）方法，在强化学习训练过程中跳过奖励模型训练、直接用现成大模型打分，结果反而优于传统RLAIF流程。更有意思的是，即便充当"标注者"的AI模型和被训练的策略模型是同一个检查点，RLAIF依然能显著超越监督微调基线，初步验证了大模型"自我提升"的可行性。这项工作为突破RLHF对大规模人工标注的依赖提供了切实可行的解决路径，对降低对齐成本、推动模型持续迭代具有重要意义。
原文 arXiv:2309.00267;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2309.00267v3