Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning
Alexandre Tamborrino , Nicola Pellicanò , Baptiste Pannier , Pascal Voitot, Louise Naudin Samsung Strategy and Innovation Center Equal contribution.
Abstract
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks Devlin et al. (2019). Most of the existing approaches rely on a randomly initialized classifier on top of such networks. We argue that this fine-tuning procedure is sub-optimal as the pre-trained model has no prior on the specific classifier labels, while it might have already learned an intrinsic textual representation of the task. In this paper, we introduce a new scoring method that casts a plausibility ranking task in a full-text format and leverages the masked language modeling head tuned during the pre-training phase. We study commonsense reasoning tasks where the model must rank a set of hypotheses given a premise, focusing on the COPA Gordon et al. (2012), Swag Zellers et al. (2018), HellaSwag Zellers et al. (2019) and CommonsenseQA Talmor et al. (2019) datasets. By exploiting our scoring method without fine-tuning, we are able to produce strong baselines (e.g. 80% test accuracy on COPA) that are comparable to supervised approaches. Moreover, when fine-tuning directly on the proposed scoring function, we show that our method provides a much more stable training phase
中文速览
预训练语言模型在常识推理任务上通常靠在顶层加一个随机初始化的分类器来做微调，但这种做法让模型对任务标签毫无先验认知，白白浪费了预训练阶段已经习得的语言知识。研究者提出了一种叫做序列评分方法（Sequence Scoring Method, SSM）的新思路：把前提和假设拼成自然语言全文，然后直接借用预训练时已经调好的掩码语言模型（Masked Language Modeling, MLM）头来打分，通过逐词遮掩前提并观察模型恢复置信度来判断哪个假设最合理。在不做任何微调的零样本设置下，该方法在COPA数据集上就达到了80%的测试准确率，直逼有监督方法的水平；在进行微调时，训练过程比标准分类器稳定得多，随机重启带来的准确率标准差降低了约10倍，而且在小数据场景下优势更为突出。这项工作表明，充分利用预训练模型内置的语言理解能力、而非依赖随机初始化的分类头，是提升常识推理任务效果与稳定性的更优路径。
原文 arXiv:2004.14074;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.14074v1