ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generation
Yu Sun Equal Contribution Shuohuan Wang∗ Shikun Feng∗ Siyu Ding Chao Pang Junyuan Shang Jiaxiang Liu Xuyi Chen Yanbin Zhao Yuxiang Lu Weixin Liu Zhihua Wu Weibao Gong Jianzhong Liang Zhizhou Shang Peng Sun Wei Liu Xuan Ouyang Dianhai Yu Hao Tian Hua Wu Haifeng Wang Baidu Inc. {sunyu02, wangshuohuan,
Abstract
Pre-trained models have achieved state-of-the-art results in various Natural Language Processing (NLP) tasks. Recent works such as T5 [1] and GPT-3 [2] have shown that scaling up pre-trained language models can improve their generalization abilities. Particularly, the GPT-3 model with 175 billion parameters shows its strong task-agnostic zero-shot/few-shot learning capabilities. Despite their success, these large-scale models are trained on plain texts without introducing knowledge such as linguistic knowledge and world knowledge. In addition, most large-scale models are trained in an auto-regressive way. As a result, this kind of traditional fine-tuning approach demonstrates relatively weak performance when solving downstream language understanding tasks. In order to solve the above problems, we propose a unified framework named ERNIE 3.0 for pre-training large-scale knowledge enhanced models. It fuses auto-regressive network and auto-encoding network, so that the trained model can be easily tailored for both natural language understanding and generation tasks with zero-shot learning, few-shot learning or fine-tuning. We trained the model with 10 billion parameters on a 4TB corpus
中文速览
大规模预训练语言模型（如GPT-3）虽然参数量惊人，但只在纯文本上训练，缺乏语言知识和世界知识，而且单纯的自回归架构在语言理解任务上表现有限。为此，百度提出了ERNIE 3.0统一预训练框架，将自回归网络（擅长生成）与自编码网络（擅长理解）融合在一起，同时在4TB的纯文本加大规模知识图谱语料上训练了一个100亿参数的模型，让它既能处理自然语言理解任务，也能处理自然语言生成任务。实验结果表明，ERNIE 3.0在54项中文NLP任务上全面超越此前最优模型，其英文版本还在SuperGLUE榜单上以90.6%的成绩超过人类基准0.8个百分点，登上榜首。这项工作证明了将知识图谱与超大规模预训练相结合、并同时兼顾理解与生成能力的路线切实可行，为后续知识增强大模型的研究提供了重要参考。
原文 arXiv:2107.02137;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2107.02137v1