A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT
Ce Zhou1111The authors contributed equally to this research. Correspondence to Ce and Qian Li Qian Li2∗ Chen Li2∗ Jun Yu3∗ Yixin Liu3∗ Guangjing Wang1 Kai Zhang3 Cheng Ji2 Qiben Yan1 Lifang He3 Hao Peng2 Jianxin Li2 Jia Wu4 Ziwei Liu5 Pengtao Xie6 Caiming Xiong7 Jian Pei8 Philip S. Yu9 Lichao Sun3 1Michigan State University, 2Beihang University, 3Lehigh University, 4Macquarie University, 5Nanyang Technological University, 6University of California San Diego, 7Salesforce AI Research,8Duke University, 9University of Illinois at Chicago
Abstract
Pretrained Foundation Models (PFMs) are regarded as the foundation for various downstream tasks with different data modalities. A PFM (e.g., BERT, ChatGPT, and GPT-4) is trained on large-scale data which provides a reasonable parameter initialization for a wide range of downstream applications. In contrast to earlier approaches that utilize convolution and recurrent modules to extract features, BERT learns bidirectional encoder representations from Transformers, which are trained on large datasets as contextual language models. Similarly, the Generative Pretrained Transformer (GPT) method employs Transformers as the feature extractor and is trained using an autoregressive paradigm on large datasets. Recently, ChatGPT shows promising success on large language models, which applies an autoregressive language model with zero shot or few shot prompting. The remarkable achievements of PFM have brought significant breakthroughs to various fields of AI in recent years. Numerous studies have proposed different methods, datasets, and evaluation metrics, raising the demand for an updated survey.
中文速览
大规模预训练基础模型（Pretrained Foundation Models, PFMs）如BERT、GPT系列和ChatGPT已成为人工智能各领域的核心技术，但学界至今缺乏一份横跨自然语言处理、计算机视觉和图学习三大领域的系统性综述。这篇论文通过梳理数百项研究，全面回顾了PFMs的基础架构（以Transformer为核心）、主流预训练方法、多模态与统一模型，以及模型压缩、安全与隐私等关键议题，并重点追踪了预训练思想从NLP向CV和图学习迁移的演进脉络。综述发现，PFMs凭借"大规模预训练＋轻量微调"的范式在几乎所有下游任务中展现出显著优势，而ChatGPT等模型引入的人类反馈强化学习和指令对齐技术正推动这一范式向通用人工智能方向跃升。这项工作的价值在于为研究者提供了一张完整的技术地图，并明确指出可扩展性、跨域泛化、逻辑推理和用户交互能力是未来PFMs研究最值得攻关的方向。
原文 arXiv:2302.09419;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2302.09419v3