PanGu-α𝛼\alpha: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computation
Wei Zeng Equal Contribution Xiaozhe Ren∗ Teng Su∗ Hui Wang∗ Yi Liao Zhiwei Wang Xin Jiang Zhenzhang Yang Kaisheng Wang Xiaoda Zhang Chen Li Ziyan Gong Yifan Yao Xinjing Huang Jun Wang Jianfeng Yu Qi Guo Yue Yu Yan Zhang Jin Wang Hengtao Tao Dasen Yan Zexuan Yi Fang Peng Fangqing Jiang Han Zhang Lingfeng Deng Yehong Zhang Zhe Lin Chao Zhang Shaojie Zhang Mingyue Guo Shanzhi Gu Gaojun Fan Yaowei Wang Xuefeng Jin Qun Liu Yonghong Tian PanGu-α𝛼\alpha Team
Abstract
Large-scale Pretrained Language Models (PLMs) have become the new paradigm for Natural Language Processing (NLP). PLMs with hundreds of billions parameters such as GPT-3 [1] have demonstrated strong performances on natural language understanding and generation with few-shot in-context learning. In this work, we present our practice on training large-scale autoregressive language models named PanGu- $\alpha$ , with up to 200 billion parameters. PanGu- $\alpha$ is developed under the MindSpore111https://www.mindspore.cn/en and trained on a cluster of 2048 Ascend 910 AI processors222https://e.huawei.com/en/products/servers/ascend. The training parallelism strategy is implemented based on MindSpore Auto-parallel, which composes five parallelism dimensions to scale the training task to 2048 processors efficiently, including data parallelism, op-level model parallelism, pipeline model parallelism, optimizer model parallelism and rematerialization. To enhance the generalization ability of PanGu- $\alpha$ , we collect 1.1TB high-quality Chinese data from a wide range of domains to pretrain the model. We empirically test the generation ability of PanGu- $\alpha$ in various scenarios includi
中文速览
中文大规模预训练语言模型长期被英文主导，缺乏能与GPT-3比肩的中文版本。为此，研究团队在华为昇腾910处理器集群上，借助MindSpore框架及五维并行训练策略，训练出了最大参数量达2000亿的中文自回归语言模型盘古α（PanGu-α），并从网页、百科、新闻、电子书等多源头精心清洗出1.1TB高质量中文语料。模型在数据摘要、问答、对话等多种生成任务上表现出色，且在零样本和少样本设置下，随着模型规模增大，各类中文NLP任务的性能也持续提升。这项工作不仅填补了超大规模中文预训练模型的空白，其并行训练框架也已开源，为学界和业界自主训练大模型提供了可复用的基础设施。
原文 arXiv:2104.12369;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.12369v1