Textbooks Are All You Need
Suriya Gunasekar Yi Zhang Jyoti Aneja Caio César Teodoro Mendes Allie Del Giorno Sivakanth Gopi Mojan Javaheripi Piero Kauffmann Gustavo de Rosa Olli Saarikivi Adil Salim Shital Shah Harkirat Singh Behl Xin Wang Sébastien Bubeck Ronen Eldan Adam Tauman Kalai Yin Tat Lee Yuanzhi Li
Abstract
We introduce phi-1, a new large language model for code, with significantly smaller size than competing models: phi-1 is a Transformer-based model with $1.3$ B parameters, trained for $4$ days on $8$ A100s, using a selection of “textbook quality” data from the web ( $6$ B tokens) and synthetically generated textbooks and exercises with GPT-3.5 ( $1$ B tokens). Despite this small scale, phi-1 attains pass@1 accuracy $50.6\%$ on HumanEval and $55.5\%$ on MBPP. It also displays surprising emergent properties compared to phi-1-base, our model before our finetuning stage on a dataset of coding exercises, and phi-1-small, a smaller model with 350M parameters trained with the same pipeline as phi-1 that still achieves $45\%$ on HumanEval.
中文速览
用不到70亿词元的"教科书级"数据训练一个13亿参数的小模型，竟然在代码生成竞赛中打败了体量大得多的对手——这正是phi-1这项工作想证明的事。研究者的核心思路是：与其堆砌海量低质量代码，不如用GPT-4筛选网络上真正有教学价值的代码片段，再用GPT-3.5批量生成类似编程教材和练习题的合成数据，让模型从"好教材"而非"杂乱代码仓库"中学习。最终，phi-1在代码生成标准测试集HumanEval上达到50.6%的pass@1准确率，在MBPP上达到55.5%，与参数量大几十倍、训练数据多几百倍的模型相当甚至更优。这项工作的重要意义在于：它用实验证明了数据质量对大语言模型的影响远超人们此前的认知，高质量数据可以从根本上改变模型的规模扩展规律，从而大幅降低训练成本和碳排放，为未来更高效地构建专用语言模型提供了一条新路径。
原文 arXiv:2306.11644;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.11644v2