Pre-trained Models for Natural Language Processing: A SurveyPTMs are also known as pre-trained language models (PLMs). In this survey, we use PTMs for NLP instead of PLMs to avoid confusion with the narrow concept of probabilistic (or statistical) language models.
Xipeng Qiu Tianxiang Sun Yige Xu Yunfan Shao Ning Dai Xuanjing Huang Address: School of Computer Science, Fudan University, Shanghai 200433, China Address: Shanghai Key Laboratory of Intelligent Information Processing, Shanghai 200433, China
Abstract
Recently, the emergence of pre-trained models (PTMs) has brought natural language processing (NLP) to a new era. In this survey, we provide a comprehensive review of PTMs for NLP. We first briefly introduce language representation learning and its research progress. Then we systematically categorize existing PTMs based on a taxonomy from four different perspectives. Next, we describe how to adapt the knowledge of PTMs to downstream tasks. Finally, we outline some potential directions of PTMs for future research. This survey is purposed to be a hands-on guide for understanding, using, and developing PTMs for various NLP tasks.
原文 arXiv:2003.08271;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2003.08271v4