Pre-training Text-to-Text Transformers for Concept-centric Common Sense

中文速览

现有预训练语言模型主要学习词语共现和句子填空，缺少对日常概念之间关系及其组合方式的常识推理能力，因而容易在问答和生成中犯错。研究者提出概念感知语言模型（Concept-Aware Language Model，CALM），通过“根据打乱的概念写回句子”（C2S）、“纠正概念顺序”（COR）和对比学习，让模型区分符合常识的句子与干扰句，并进一步用模型自己生成的句子互相训练。实验显示，只需在较小语料上进行少量增量预训练，CALM就在CommonsenseQA、OpenBookQA、PIQA、aNLI和CommonGEN等理解与生成任务上稳定超过T5-base，表现甚至接近更大的模型。它的重要性在于无需外部知识图谱或改动模型结构，就能以“即插即用”的方式增强预训练模型的生成和判别式常识推理能力。

摘要

Pre-trained language models (PTLM) have achieved impressive results in a range of natural language understanding (NLU) and generation (NLG) tasks. However, current pre-training objectives such as masked token prediction (for BERT-style PTLMs) and masked span infilling (for T5-style PTLMs) do not explicitly model the relational commonsense knowledge about everyday concepts, which is crucial to many downstream tasks that need common sense to understand or generate. To augment PTLMs with concept-centric commonsense knowledge, in this paper, we propose both generative and contrastive objectives for learning common sense from the text, and use them as intermediate self-supervised learning tasks for incrementally pre-training PTLMs (before task-specific fine-tuning on downstream datasets). Furthermore, we develop a joint pre-training framework to unify generative and contrastive objectives so that they can mutually reinforce each other. Extensive experimental results show that our method, concept-aware language model (CALM)11 1 Code and data have been uploaded and will be published: https://anonymous.4open.science/repository/6fdeed55-ec2c-4ffa-aee8-0cc3b7f5ade5, can pack more commonsense knowledge into the parameters of a pre-trained text-to-text transformer without relying on external knowledge graphs, yielding better performance on both NLU and NLG tasks. We show that while only incrementally pre-trained on a relatively small corpus for a few steps, CALM outperforms baseline methods by a consistent margin and even comparable with some larger PTLMs, which suggests that CALM can serve as a general, “plug-and-play” method for improving the commonsense reasoning ability of a PTLM.

术语表

Pre-trained Language Model (PTLM)

预训练语言模型（PTLM）

natural language understanding (NLU)

自然语言理解（NLU）

natural language generation (NLG)

自然语言生成（NLG）

masked token prediction

掩码词元预测

masked span infilling

掩码跨度填充

commonsense knowledge

常识知识

concept-centric commonsense knowledge

以概念为中心的常识知识

commonsense reasoning

常识推理

generative objective

生成式目标

contrastive objective

对比式目标

self-supervised learning

自监督学习

incremental pre-training

增量式预训练

joint pre-training framework

联合预训练框架

Concept-Aware Language Model (CALM)

概念感知语言模型（CALM）

BERT

BERT

T5

T5

RoBERTa

RoBERTa

ALBERT

ALBERT

BART

BART

UnifiedQA

UnifiedQA

concept-to-sentence generation (C2S)

概念到句子生成（C2S）

concept order recovering (COR)

概念顺序恢复（COR）

intermediate-task transfer learning

中间任务迁移学习

text-to-text transformer

文本到文本转换器

concept extraction

概念抽取

part-of-speech tagging

词性标注

CommonsenseQA

CommonsenseQA（常识问答数据集）

OpenBookQA

OpenBookQA（开放书本问答数据集）

PIQA

PIQA（物理常识问答数据集）

CommonGEN

CommonGEN（常识生成数据集）