Improving and Simplifying Pattern Exploiting Training
Derek Tam Equal contribution Rakesh R Menon \ANDMohit Bansal Shashank Srivastava Colin Raffel UNC Chapel Hill {dtredsox, rrmenon, mbansal, ssrivastava,
Abstract
Recently, pre-trained language models (LMs) have achieved strong performance when fine-tuned on difficult benchmarks like SuperGLUE. However, performance can suffer when there are very few labeled examples available for fine-tuning. Pattern Exploiting Training (Pet) is a recent approach that leverages patterns for few-shot learning. However, Pet uses task-specific unlabeled data. In this paper, we focus on few shot learning without any unlabeled data and introduce ADAPET, which modifies Pet’s objective to provide denser supervision during fine-tuning. As a result, ADAPET outperforms Pet on SuperGLUE without any task-specific unlabeled data. Our code can be found at https://github.com/rrmenon10/ADAPET.
中文速览
标注数据极度稀缺时，如何让中小规模语言模型也能学得好，是个现实难题。研究者在已有的"模式利用训练"方法（Pattern Exploiting Training，PET）基础上提出了ADAPET，核心改进有两点：一是把标签损失解耦，让模型对所有词汇的预测都产生梯度信号，而不只盯着候选标签；二是引入"标签条件遮蔽语言模型"目标，让模型在给定正确或错误标签的前提下反向预测输入文本，从而获得更密集的监督信号。在每个任务仅有32条标注样本的SuperGLUE基准上，ADAPET不依赖任何任务专属无标注数据，却在验证集和测试集上均超越了需要大量无标注数据的iPet，也以仅GPT-3参数量0.1%的规模超过了GPT-3，为小模型少样本学习提供了一条更实用的路径。
原文 arXiv:2103.11955;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.11955v3