Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models
Zixiang Chen Yihe Deng Huizhuo Yuan Kaixuan Ji Quanquan Gu
Abstract
Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and da
中文速览
无需额外收集人工标注数据，仅凭已有的监督微调（SFT）数据集，就能让一个较弱的大语言模型（LLM）持续变强——这篇论文提出了一种名为"自我博弈微调"（SPIN，Self-Play fIne-tuNing）的方法来实现这一目标。核心思路是让模型与"上一版本的自己"对弈：每轮迭代中，旧版模型生成回答作为"负样本"，人工标注的回答作为"正样本"，新版模型通过学习区分这两类回答来提升自身，如此循环直到模型输出分布逼近真实数据分布。实验表明，基于 Mistral-7B 的 zephyr-7b-sft-full 模型经过 SPIN 迭代后，在 HuggingFace Open LLM 排行榜上平均分从 58.14 提升到 63.16，在 MT-Bench 上从 5.94 提升到 6.78，甚至可以媲美额外使用了 6.2 万条 GPT-4 偏好数据训练的 DPO 模型。这项工作的意义在于：它证明了在不借助更强外部模型或额外人工反馈的前提下，自我博弈机制本身就足以充分挖掘现有标注数据的潜力，为低成本、可扩展的 LLM 对齐提供了新思路。
原文 arXiv:2401.01335;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2401.01335v3