A Critical Evaluation of AI Feedback for Aligning Large Language Models
Archit Sharma Sedrick Keh Eric Mitchell Chelsea Finn Kushal Arora Thomas Kollar
Abstract
Reinforcement learning with AI feedback (RLAIF) is a popular paradigm for improving the instruction-following abilities of powerful pre-trained language models. RLAIF first performs supervised fine-tuning (SFT) using demonstrations from a teacher model and then further fine-tunes the model with reinforcement learning (RL), using feedback from a critic model. While recent popular open-source models have demonstrated substantial improvements in performance from the RL step, in this paper we question whether the complexity of this RL step is truly warranted for AI feedback. We show that the improvements of the RL step are virtually entirely due to the widespread practice of using a weaker teacher model (e.g. GPT-3.5) for SFT data collection than the critic (e.g., GPT-4) used for AI feedback generation. Specifically, we show that simple supervised fine-tuning with GPT-4 as the teacher outperforms existing RLAIF pipelines. More generally, we find that the gains from RLAIF vary substantially across base model families, test-time evaluation protocols, and critic models. Finally, we provide a mechanistic explanation for when SFT may outperform the full two-step RLAIF pipeline as well as su
中文速览
用GPT-3.5生成示范数据、再用GPT-4的偏好反馈做强化学习（RLAIF）是当前微调开源大语言模型的流行范式，但这套流程真的有效吗？研究者通过系统对比发现，RLAIF看似优于监督微调（SFT）的性能增益，其实几乎完全源于"教师模型弱（GPT-3.5）、批评模型强（GPT-4）"这一能力错配——一旦直接用GPT-4生成SFT训练数据，简单的监督微调就能持平甚至超过完整的RLAIF流程。进一步实验表明，RLAIF的收益还高度依赖基础模型的能力、评估协议和批评模型的选择，并不稳定。作者还在一个简化的赌博机（bandit）理论框架下给出了机制性解释，并提出了让RLAIF真正发挥作用的实践建议。这一发现警示研究者：现有基准可能系统性地高估了AI反馈在指令跟随上的贡献，未来应优先考虑使用更强教师模型的SFT数据集，并重新审视RLAIF流程的真实价值。
原文 arXiv:2402.12366;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2402.12366v1