Aha.
正在载入中英对照阅读…

arXiv:2401.01335 · 中英对照阅读

自博弈微调将弱语言模型转变为强语言模型

Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

Zixiang Chen、Yihe Deng、Huizhuo Yuan、Kaixuan Ji、Quanquan Gu

中文速览

让一个已经用人工示范数据微调过的弱模型继续变强,往往还得花钱收集更多人工标注或偏好反馈;SPIN 想解决的正是这个难题。它让模型反复和自己的旧版本“过招”:旧版本根据同一批问题生成回答,新版本学习区分这些回答与人工示范,并逐轮调整,让自己的输出更接近人工数据。实验中,SPIN 在多项语言模型基准测试上持续提升表现,部分成绩超过了使用额外 GPT-4 偏好数据训练的模型;理论分析也表明,训练目标的最优点对应模型输出与目标数据分布一致。这个结果说明,经过巧妙的自我对照,有限的人工示范数据也能被更充分地利用,减少模型变强对额外标注和强力外部教师的依赖。

摘要

借助监督微调(SFT)发挥人工标注数据的效用,是推进大语言模型(LLM)发展的关键。本文深入探讨如何在无需获取额外人工标注数据的情况下,将一个较弱的大语言模型培养成强大的模型。我们提出一种名为自博弈微调(SPIN)的新型微调方法,该方法以一个经过监督微调的模型为起点。SPIN 的核心是自博弈机制,大语言模型通过与自身的不同实例对弈来提升能力。具体而言,大语言模型根据先前迭代生成自己的训练数据,并通过辨别这些自生成响应与人工标注数据所获得的响应来优化自身策略。我们的方法逐步将大语言模型从初生模型提升为强大的模型,充分释放人工标注示范数据用于 SFT 的潜力。从理论上看,我们证明,只有当大语言模型策略与目标数据分布对齐时,才能达到本方法训练目标函数的全局最优解。在实证方面,我们在多个基准数据集上评估了本方法,包括 HuggingFace Open LLM Leaderboard、MT-Bench 以及来自 Big-Bench 的数据集。结果表明,SPIN 能够显著提升大语言模型在多种基准上的表现,甚至优于使用额外 GPT-4 偏好数据进行补充训练的直接偏好优化(DPO)模型。这凸显了自博弈的潜力:无需专家对手,也能使大语言模型达到人类水平的表现。代码可在 https://github.com/uclaml/SPIN 获取。

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM’s performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

术语表

Large Language Models (LLMs)
大语言模型(LLM)
Supervised Fine-Tuning (SFT)
监督微调(SFT)
Self-Play fIne-tuNing (SPIN)
自博弈微调(SPIN)
human-annotated data
人工标注数据
human demonstrations
人工示范数据
self-play
自博弈
self-training
自训练
weak-to-strong learning
由弱到强学习
alignment
对齐
Reinforcement Learning from Human Feedback (RLHF)
基于人类反馈的强化学习(RLHF)
direct preference optimization (DPO)
直接偏好优化(DPO)
generative adversarial networks (GAN)
生成对抗网络(GAN)
LLM policy
大语言模型策略
target data distribution
目标数据分布
training objective function
训练目标函数
global optimum
全局最优解
prompt
提示
response generation
响应生成
synthetic data
合成数据
binary feedback
二元反馈
reward model
奖励模型
boosting algorithms
提升算法
AlphaGo Zero
AlphaGo Zero
AlphaZero
AlphaZero
HuggingFace Open LLM Leaderboard
HuggingFace Open LLM Leaderboard
MT-Bench
MT-Bench
Big-Bench
Big-Bench
Ultrachat200k
Ultrachat200k
GSM8k
GSM8k
TruthfulQA
TruthfulQA