Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judge
Tianhao Wu1,2 Weizhe Yuan1,3 Olga Golovneva1 Jing Xu1 Yuandong Tian1 Jiantao Jiao2 Jason Weston1,3 Sainbayar Sukhbaatar1 1Meta FAIR 2University of California, Berkeley 3New York University
Abstract
Large Language Models (LLMs) are rapidly surpassing human knowledge in many domains. While improving these models traditionally relies on costly human data, recent self-rewarding mechanisms (Yuan et al., 2024c) have shown that LLMs can improve by judging their own responses instead of relying on human labelers. However, existing methods have primarily focused on improving model responses rather than judgment capabilities, resulting in rapid saturation during iterative training. To address this issue, we introduce a novel Meta-Rewarding step to the self-improvement process, where the model judges its own judgements and uses that feedback to refine its judgment skills. Surprisingly, this unsupervised approach improves the model’s ability to judge and follow instructions, as demonstrated by a win rate improvement of Llama-3-8B-Instruct from 22.9% to 39.4% on AlpacaEval 2, and 20.6% to 29.1% on Arena-Hard. These results strongly suggest the potential for self-improving models without human supervision.
中文速览
大型语言模型（LLM）在迭代训练中普遍面临"自我奖励"机制快速饱和的问题——模型一直在练习"写回答"，却从不提升"打分"本身的能力，导致评判越来越不准、训练越来越没用。为此，研究者提出了「元奖励（Meta-Rewarding）」方法：在原有"演员-评判者"的双角色自博弈框架上，再引入第三个角色"元评判者"，让模型对自己打出的分数再做一轮评估，从而同时优化"写答案"和"打分"两项能力，整个过程无需任何额外的人工标注数据。以 Llama-3-8B-Instruct 为起点经过四轮迭代训练后，模型在 AlpacaEval 2 上的长度控制胜率从 22.9% 跃升至 39.4%，在 Arena-Hard 上也从 20.6% 提升到 29.1%，甚至超越了 GPT-4-0314。这一结果表明，通过让模型"反思自己的判断"来同步强化评判能力，可以显著突破自我改进的瓶颈，为在没有人类监督的情况下持续提升 AI 能力提供了一条可行路径。
原文 arXiv:2407.19594;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2407.19594v2