Knowledge Graph Based Synthetic Corpus Generation for Knowledge-Enhanced Language Model Pre-training
Oshin Agarwal 1 Heming Ge2 Siamak Shakeri2 Rami Al-Rfou2 1 University of Pennsylvania 2Google Research {hemingge, siamaks, Work done during internship at Google
Abstract
Prior work on Data-To-Text Generation, the task of converting knowledge graph (KG) triples into natural text, focused on domain-specific benchmark datasets. In this paper, however, we verbalize the entire English Wikidata KG, and discuss the unique challenges associated with a broad, open-domain, large-scale verbalization. We further show that verbalizing a comprehensive, encyclopedic KG like Wikidata can be used to integrate structured KGs and natural language corpora. In contrast to the many architectures that have been developed to integrate these two sources, our approach converts the KG into natural text, allowing it to be seamlessly integrated into existing language models. It carries the further advantages of improved factual accuracy and reduced toxicity in the resulting language model. We evaluate this approach by augmenting the retrieval corpus in a retrieval language model and showing significant improvements on the knowledge intensive tasks of open domain QA and the LAMA knowledge probe.
中文速览
知识图谱（Knowledge Graph, KG）拥有海量结构化事实，但语言模型只能直接消化自然语言文本，两者之间存在格式鸿沟。研究者们开发了名为 TeKGen 的序列到序列模型，通过远程监督对齐、两阶段微调 T5-large 以及语义质量过滤，将整个英文 Wikidata 知识图谱（约 4500 万条三元组、1500 种关系）转化为约 1800 万条自然语言句子，形成 KeLM 语料库。他们将这份合成语料库补充进检索式语言模型 REALM 的检索库，在开放域问答和 LAMA 知识探测两项测试上均取得显著提升，同时还发现文本化后的模型比直接使用原始三元组具有更高的事实准确性和更低的毒性。这项工作提供了一条将结构化知识图谱无缝融入现有语言模型的简洁路径，对构建事实更可靠、偏见更少的大规模语言模型具有重要参考价值。
原文 arXiv:2010.12688;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.12688v2