Pre-training Text-to-Text Transformers for Concept-centric Common Sense
Wangchunshu Zhou1, Dong-Ho Lee2∗, Ravi Kiran Selvam2, Seyeon Lee2, Bill Yuchen Lin2, Xiang Ren2 1 Beihang University 2 University of Southern California {dongho.lee, Equal contribution. The work was done when Wangchunshu was visiting USC.
Abstract
Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centric commonsense knowledge, in this paper, we propose both generative and contrastive objectives for learning common sense from the text, and use them as intermediate self-supervised learning tasks for incrementally pre-training PTLMs (before task-specific fine-tuning on downstream datasets). Furthermore, we develop a joint pre-training framework to unify generative and contrastive objectives so that they can mutually reinforce each other. Extensive experimental results show that our method, concept-aware language model (CALM)111Code and data have been uploaded and will be published: https://anonymous.4open.science/repository/6fdeed55-ec2c-4ffa-aee8-0cc3b7f5ade5, can pack more commonsense k
中文速览
预训练语言模型（如BERT、T5）虽然在各类自然语言理解与生成任务中表现出色，却缺乏对日常概念间关系性常识（relational commonsense knowledge）的显式建模，导致在需要常识推理的任务上频繁出错。为此，本文提出了一种名为CALM（concept-aware language model，概念感知语言模型）的方法，设计了两类自监督预训练目标：概念到句子生成（C2S，让模型从一组无序关键词重建合理句子）和概念顺序恢复（COR，让模型纠正概念顺序被打乱的句子），并进一步引入对比学习目标，让模型区分真实句子与由自身生成的"干扰句"，三者通过联合训练框架相互强化。实验表明，仅在小规模语料上进行少量增量预训练，CALM在四个常识理解数据集（CommonsenseQA、OpenBookQA、PIQA、aNLI）和常识生成数据集CommonGEN上均稳定超越T5-base基线，甚至媲美部分更大规模的预训练模型。这项工作的价值在于，它无需依赖外部知识图谱，便能以"即插即用"的方式为任意文本到文本的预训练模型注入更丰富的常识推理能力。
原文 arXiv:2011.07956;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.07956v2