Uncertainty Estimation for Language Reward Models
Adam Gleave1 Geoffrey Irving2 Work conducted during an internship at DeepMind.
Abstract
Language models can learn a range of capabilities from unsupervised training on text corpora. However, to solve a particular problem (such as text summarization) it is typically necessary to fine-tune them on a task-specific dataset. It is often easier for humans to choose between options than to provide labeled data, and prior work has achieved state-of-the-art performance by training a reward model from such preference comparisons. However, collecting a large preference comparison dataset is still expensive—and the learned reward models are unreliable out-of-distribution. We seek to address these problems via uncertainty estimation, which can improve sample efficiency and robustness using active learning and risk-averse reinforcement learning (RL). Specifically, we use bootstrap aggregating (bagging) to train an ensemble of reward models differing in the initialization of their final layer. Ensembles have proved successful in prior applications of active learning [9, 3], but we find that in our setting ensemble active learning does not outperform random sampling. Further experiments show that while the aggregate predictions are well-calibrated, the ensemble’s estimated epistemic
中文速览
用人类偏好对比数据（preference comparisons）来训练奖励模型、再用强化学习微调语言模型，是当前让模型输出更符合人类期望的主流路线，但这条路线需要大量标注数据，且奖励模型在训练分布之外容易失效。研究者尝试用集成学习（ensemble）方法为奖励模型估计不确定性，希望借此实现主动学习（active learning）以减少标注量，同时让强化学习训练更稳健。实验结果令人失望：基于集成的主动学习并不比随机采样更高效，集成模型估计的认知不确定性（epistemic uncertainty）与模型实际误差的相关性也很弱。作者认为根本原因在于所有集成成员都从同一个预训练模型微调而来，彼此太过相似，多样性不足以支撑有效的不确定性估计——这一发现对"训练单一大型预训练模型再微调"的主流范式提出了挑战，暗示未来可能需要训练多个独立的预训练模型，或通过其他方式在预训练阶段就引入多样性。
原文 arXiv:2203.07472;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2203.07472v1