Instruction Tuning with GPT-4
Baolin Peng∗*, Chunyuan Li∗*, Pengcheng He∗*, Michel Galley, Jianfeng Gao Microsoft Research
Abstract
Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available. ∗Equal Contribution 111https://instruction-tuning-with-gpt-4.github.io/ Note: This is a preliminary release, and we will continue to expand the dataset and will finetune larger models.
中文速览
用GPT-4生成的指令跟随数据（instruction-following data）来微调开源大语言模型LLaMA，能显著提升模型在全新任务上的零样本表现——这正是本文要解决的核心问题。研究者复用了Alpaca的5.2万条指令，改用GPT-4而非GPT-3.5来生成英文和中文的回答，同时还收集了GPT-4对多个模型输出的打分与比较数据，用于训练奖励模型。实验表明，用GPT-4数据微调后的7B参数LLaMA，在帮助性等对齐指标上远超用GPT-3.5数据训练的Alpaca（13B），且与GPT-4本身的表现相当接近。这项工作意义在于：它证明了借助GPT-4这位"教师"，低成本地让开源小模型获得接近顶级商业模型的指令跟随能力是可行的，相关数据和代码也已公开，为后续开源社区研究提供了重要基础。
原文 arXiv:2304.03277;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.03277v1