arXiv:2210.10760 · 中英对照阅读
奖励模型过度优化的缩放定律
Scaling Laws for Reward Model Overoptimization
中文速览
当语言模型被训练着追求一个不够准确的“奖励分数”时,优化得越狠,真实表现可能反而越差;论文想弄清这种“指标被钻空子”的现象有多严重、会怎样随模型规模变化。作者用一个大型奖励模型充当“标准答案”,据此生成偏好数据来训练较小的代理模型,再分别用强化学习和“生成多个答案、挑分数最高的”(best-of-n)来优化语言模型。实验发现,两种方法下真实得分都会先升后降,但变化规律不同;奖励模型更大、训练数据更多通常更有帮助,而更大的语言模型并没有表现出更严重的过度优化,强化学习也比 best-of-n 更“费”策略变化。结果为估算奖励模型能被安全优化到什么程度提供了实证依据,也提醒我们:AI若过度追逐一个有偏差的指标,可能离真正想要的目标越来越远。
摘要
在基于人类反馈的强化学习中,通常会针对一个经训练以预测人类偏好的奖励模型进行优化。由于奖励模型只是一个不完美的代理,过度优化其数值可能会损害真实表现,这符合古德哈特定律。人们经常观察到这种效应,但由于收集人类偏好数据的成本高昂,尚未对其进行仔细测量。在本研究中,我们采用一种合成实验设置,其中一个固定的“黄金标准”奖励模型扮演人类的角色,提供用于训练代理奖励模型的标签。我们研究了在使用强化学习或最佳 n 采样针对代理奖励模型进行优化时,黄金标准奖励模型的分数如何变化。我们发现,这种关系的函数形式因优化方法而异;在两种情况下,其系数都会随着奖励模型参数数量的增加而平滑缩放。我们还研究了奖励模型数据集的规模、奖励模型和策略的参数数量,以及强化学习设置中加到奖励上的 KL 惩罚项系数对这种关系的影响。最后,我们探讨了这些实证结果对 AI 对齐理论考量的意义。
In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed “gold-standard” reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of- $n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of these empirical results for theoretical considerations in AI alignment.
术语表
- Reinforcement Learning from Human Feedback (RLHF)
- 基于人类反馈的强化学习
- reward model (RM)
- 奖励模型
- human preference
- 人类偏好
- Goodhart’s law
- 古德哈特定律
- proxy objective
- 代理目标
- overoptimization
- 过度优化
- gold-standard reward model
- 黄金标准奖励模型
- synthetic setup
- 合成实验设置
- best-of-n (BoN) sampling
- 最佳 n 采样
- rejection sampling
- 拒绝采样
- reranking
- 重排序
- policy gradient
- 策略梯度
- Kullback–Leibler (KL) divergence
- Kullback–Leibler 散度
- KL penalty
- KL 惩罚项
- scaling law
- 缩放定律
- Proximal Policy Optimization (PPO)
- 近端策略优化
- supervised fine-tuning (SFT)
- 监督微调
- GPT-3
- GPT-3
- InstructGPT
- InstructGPT
- policy
- 策略
- trajectory
- 轨迹
- rollout
- 采样轨迹
- preference label
- 偏好标签
- synthetic training data
- 合成训练数据
- AI alignment
- AI 对齐
- misalignment
- 失配
- hyperparameter
- 超参数