Zephyr: Direct Distillation of LM Alignment
Lewis Tunstall, Edward Beeching,∗ Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf The H4 (Helpful, Honest, Harmless, Huggy) Team https://huggingface.co/HuggingFaceH4 Equal contribution.
Abstract
We aim to produce a smaller language model that is aligned to user intent. Previous research has shown that applying distilled supervised fine-tuning (dSFT) on larger models significantly improves task accuracy; however, these models are unaligned, i.e. they do not respond well to natural prompts. To distill this property, we experiment with the use of preference data from AI Feedback (AIF). Starting from a dataset of outputs ranked by a teacher model, we apply distilled direct preference optimization (dDPO) to learn a chat model with significantly improved intent alignment. The approach requires only a few hours of training without any additional sampling during fine-tuning. The final result, Zephyr-7B, sets a new state-of-the-art on chat benchmarks for 7B parameter models, and requires no human annotation. In particular, results on MT-Bench show that Zephyr-7B surpasses Llama2-Chat-70B, the best open-access RLHF-based model. Code, models, data, and tutorials for the system are available at https://github.com/huggingface/alignment-handbook.
中文速览
如何用纯蒸馏方式让70亿参数的小模型也能像大模型那样"听懂人话"，是这项工作要回答的核心问题。研究者以Mistral-7B为基础，先用GPT-3.5-turbo生成的对话数据做监督微调（蒸馏式SFT，dSFT），再借助GPT-4对多个模型输出打分得到的偏好数据集UltraFeedback，用直接偏好优化（Direct Preference Optimization，DPO）对模型做进一步对齐，整个流程不需要任何人工标注，在16张A100上仅需几小时即可完成。最终得到的Zephyr-7B在MT-Bench等对话评测上不仅超越了所有同规模开源模型，甚至超过了参数量大10倍、依赖大量人工反馈训练的Llama2-Chat-70B。这项工作表明，通过巧妙利用AI反馈替代人工标注，小模型同样能够获得高质量的意图对齐能力，为资源有限的研究者提供了一条切实可行的开源对齐路径。
原文 arXiv:2310.16944;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.16944v1