Pre-trained Models for Natural Language Processing: A Survey
Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang School of Computer Science, Fudan University, Shanghai 200433, China Shanghai Key Laboratory of Intelligent Information Processing, Shanghai 200433, China
Abstract
Recently, the emergence of pre-trained models (PTMs)111PTMs are also known as pre-trained language models (PLMs). In this survey, we use PTMs for NLP instead of PLMs to avoid confusion with the narrow concept of probabilistic (or statistical) language models. has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
中文速览
预训练语言模型（Pre-trained Models, PTMs）正在深刻改变自然语言处理（NLP）的研究格局——从早期静态的词向量（如Word2Vec、GloVe），到能感知上下文的深层模型（如ELMo、BERT、GPT），模型的表达能力越来越强。这篇综述系统梳理了NLP预训练模型的发展脉络，从语言表示学习的基本原理出发，按照表示类型、模型架构、预训练任务目标和应用场景四个维度，对现有PTMs进行了全面分类。论文还详细讨论了如何通过微调（fine-tuning）、提示（prompt）等方式把预训练知识迁移到各类下游任务，并整理了开源工具、语料库等丰富资源。这项工作的重要价值在于，它为研究者和工程师提供了一份系统性的"全景地图"，帮助人们理解、使用乃至进一步开发预训练模型，同时也指出了该方向在计算效率、可解释性、跨语言泛化等方面尚待突破的挑战。
原文 arXiv:2003.08271;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.08271v4