Baichuan 2: Open Large-scale Language Models
Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Chao Yin, Chenxu Lv, Da Pan Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji Jian Xie, Juntao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yaodong Yang Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, Zhiying Wu Baichuan Inc.
Abstract
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed-source or limited in their capability for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multilingual language models containing 7 billion and 13 billion parameters, trained from scratch, on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open-source models of similar size on public benchmarks like MMLU, CMMLU, GSM8K, and HumanEval. Furthermore, Baichuan 2 excels in vertical domains such as medicine and law. We will release all pre-training model checkpoints to benefit the research community in better understanding the training dynamics of Baichuan 2. ††Authors are listed alphabetically, correspondent: daniel@baichuan-inc.com. ††Jiaming Ji, Borong Zhang, Xuehai Pan, Mickel Liu, Juntao Dai, Ruiyang Sun, Yaodong Yang affiliated with Peking University.
中文速览
现有主流大语言模型（LLM）要么闭源、要么对中文等非英语语言支持薄弱，严重制约了全球研究者的使用与改进。百川团队从零开始，用2.6万亿个token训练了70亿和130亿参数两款多语言模型Baichuan 2，并通过扩充词表、引入NormHead与Max-z Loss等技术手段稳定训练过程、提升推理鲁棒性。在MMLU、CMMLU、GSM8K、HumanEval等公开榜单上，Baichuan 2达到或超越同规模开源模型，数学与代码能力相比上一代几乎翻倍，在医疗、法律等垂直领域同样表现突出。团队不仅开放了基础模型和对话模型的全部权重，还罕见地发布了从2000亿到2.6万亿token的全程训练检查点，为研究者深入理解大模型训练动态提供了宝贵资源。
原文 arXiv:2309.10305;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2309.10305v4