Textbooks Are All You Need II: phi-1.5 technical report
Yuanzhi Li Sébastien Bubeck Ronen Eldan Allie Del Giorno Suriya Gunasekar Yin Tat Lee
Abstract
We continue the investigation into the power of smaller Transformer-based language models as initiated by TinyStories – a 10 million parameter model that can produce coherent English – and the follow-up work on phi-1, a 1.3 billion parameter model with Python coding performance close to the state-of-the-art. The latter work proposed to use existing Large Language Models (LLMs) to generate “textbook quality” data as a way to enhance the learning process compared to traditional web data. We follow the “Textbooks Are All You Need” approach, focusing this time on common sense reasoning in natural language, and create a new 1.3 billion parameter model named phi-1.5, with performance on natural language tasks comparable to models 5x larger, and surpassing most non-frontier LLMs on more complex reasoning tasks such as grade-school mathematics and basic coding. More generally, phi-1.5 exhibits many of the traits of much larger LLMs, both good –such as the ability to “think step by step” or perform some rudimentary in-context learning– and bad, including hallucinations and the potential for toxic and biased generations –encouragingly though, we are seeing improvement on that front thanks to
中文速览
13亿参数的小模型也能媲美70亿参数大模型？微软研究团队用一个关键发现做到了这点：与其用海量互联网文本喂模型，不如用大型语言模型生成"教科书级别"的高质量合成数据来训练。他们据此构建了名为 phi-1.5 的语言模型，仅有13亿参数、训练于300亿词元，却在常识推理基准测试上追平了体量大5倍的同类模型，在小学数学和基础代码生成等复杂推理任务上甚至超过了大多数非顶尖大模型，包括650亿参数的 Llama。更值得关注的是，由于训练数据几乎全为合成生成、几乎不含真实网络文本，模型产生有毒和偏见内容的倾向也明显低于同规模模型，为解决大语言模型安全问题提供了一条新思路。研究团队将 phi-1.5 完整开源，旨在让更多研究者能以较低计算成本深入探究大模型的涌现能力、幻觉机制及内容安全等迫切问题。
原文 arXiv:2309.05463;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2309.05463v1