Scaling Laws for Reward Model Overoptimization
Leo Gao OpenAI、John Schulman OpenAI、Jacob Hilton OpenAI
Abstract
In reinforcement learning from human feedback, it is common to optimize against a reward model trained to predict human preferences. Because the reward model is an imperfect proxy, optimizing its value too much can hinder ground truth performance, in accordance with Goodhart’s law. This effect has been frequently observed, but not carefully measured due to the expense of collecting human preference data. In this work, we use a synthetic setup in which a fixed “gold-standard” reward model plays the role of humans, providing labels used to train a proxy reward model. We study how the gold reward model score changes as we optimize against the proxy reward model using either reinforcement learning or best-of- $n$ sampling. We find that this relationship follows a different functional form depending on the method of optimization, and that in both cases its coefficients scale smoothly with the number of reward model parameters. We also study the effect on this relationship of the size of the reward model dataset, the number of reward model and policy parameters, and the coefficient of the KL penalty added to the reward in the reinforcement learning setup. We explore the implications of t
中文速览
人类反馈强化学习（RLHF）中，AI系统是靠一个"代理奖励模型"来学习人类偏好的，但过度优化这个代理目标反而会让真实表现下降——这正是古德哈特定律在AI里的体现，却因为收集真实人类标注数据太贵，一直缺乏精确的量化研究。为此，研究者搭建了一套合成实验框架：用一个大型"黄金奖励模型"扮演人类来提供标注，再训练规模不同的代理奖励模型，分别用强化学习（RL）和最优-N采样（best-of-n）两种方式对其进行优化，同时持续追踪黄金模型给出的真实得分。实验发现，黄金得分随策略与初始策略之间KL散度的变化遵循两种不同的函数形式——BoN呈先线性后饱和的增长，RL则先增后降并带有平方项惩罚——而且这两种形式的系数都随代理奖励模型参数量的增大呈平滑对数规律变化，使得性能预测成为可能。这项工作首次为RLHF中的过优化现象提供了可量化、可外推的缩放定律，为如何安全地优化AI系统的奖励模型、避免未来更强大模型发生危险性目标偏移提供了重要的实证基础。
原文 arXiv:2210.10760;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.10760v1