Pre-training Text-to-Text Transformers for Concept-centric Common Sense
Wangchunshu Zhou1, Dong-Ho Lee2∗, Ravi Kiran Selvam2, Seyeon Lee2, Bill Yuchen Lin2, Xiang Ren2 1 Beihang University 2 University of Southern California {dongho.lee, Equal contribution. The work was done when Wangchunshu was visiting USC.
Abstract
Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centric commonsense knowledge, in this paper, we propose both generative and contrastive objectives for learning common sense from the text, and use them as intermediate self-supervised learning tasks for incrementally pre-training PTLMs (before task-specific fine-tuning on downstream datasets). Furthermore, we develop a joint pre-training framework to unify generative and contrastive objectives so that they can mutually reinforce each other. Extensive experimental results show that our method, concept-aware language model (CALM)111Code and data have been uploaded and will be published: https://anonymous.4open.science/repository/6fdeed55-ec2c-4ffa-aee8-0cc3b7f5ade5, can pack more commonsense k
中文速览
现有预训练语言模型主要学习词语共现和句子填空，缺少对日常概念之间关系及其组合方式的常识推理能力，因而容易在问答和生成中犯错。研究者提出概念感知语言模型（Concept-Aware Language Model，CALM），通过“根据打乱的概念写回句子”（C2S）、“纠正概念顺序”（COR）和对比学习，让模型区分符合常识的句子与干扰句，并进一步用模型自己生成的句子互相训练。实验显示，只需在较小语料上进行少量增量预训练，CALM就在CommonsenseQA、OpenBookQA、PIQA、aNLI和CommonGEN等理解与生成任务上稳定超过T5-base，表现甚至接近更大的模型。它的重要性在于无需外部知识图谱或改动模型结构，就能以“即插即用”的方式增强预训练模型的生成和判别式常识推理能力。
原文 arXiv:2011.07956;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2011.07956v2