Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learning
Shaohua Wu、Xudong Zhao、Tong Yu \ANDRongguo Zhang、Chong Shen、Hongli Liu、Feng Li \ANDHong Zhu、Jiangang Luo、Liang Xu、Xuanwei Zhang Inspur Artificial Intelligence Research Institute
Abstract
Recent work like GPT-3 has demonstrated excellent performance of Zero-Shot and Few-Shot learning on many natural language processing (NLP) tasks by scaling up model size, dataset size and the amount of computation. However, training a model like GPT-3 requires huge amount of computational resources which makes it challengeable to researchers. In this work, we propose a method that incorporates large-scale distributed training performance into model architecture design. With this method, Yuan 1.0, the current largest singleton language model with 245B parameters, achieves excellent performance on thousands GPUs during training, and the state-of-the-art results on natural language processing tasks. A data processing method is designed to efficiently filter massive amount of raw data. The current largest high-quality Chinese corpus with 5TB high quality texts is built based on this method. In addition, a calibration and label expansion method is proposed to improve the Zero-Shot and Few-Shot performance, and steady improvement is observed on the accuracy of various tasks. Yuan 1.0 presents strong capacity of NLP tasks, and the generated articles are difficult to distinguish from the h
中文速览
大规模语言模型的零样本（Zero-Shot）和少样本（Few-Shot）能力极具价值，但训练成本极高，这让大多数研究者望而却步。为此，研究团队提出将大规模分布式训练效率直接纳入模型架构设计，通过对张量并行、流水线并行和数据并行三种策略的理论分析，指导层数、隐层维度、批大小等超参数的选择，最终训练出拥有2450亿参数的单体语言模型"源1.0"（Yuan 1.0），是目前规模最大的非专家混合（Singleton）语言模型。团队还构建了一套基于Spark和BERT分类器的海量数据过滤系统，整理出5TB高质量中文语料，并提出标定与标签扩展方法来稳步提升零样本和少样本任务的准确率。实验表明，Yuan 1.0在多项中文自然语言处理任务上达到最优水平，生成的文章难以与人类写作区分，同时在2128块GPU上实现了45%的理论峰值算力利用率，为国内研究者以可承受的计算成本探索超大规模语言模型提供了一条可行路径。
原文 arXiv:2110.04725;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2110.04725v2