Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling
Linyao Yang, Hongyang Chen, , Zhao Li, Xiao Ding, Xindong Wu This work was supported in part by National Natural Science Foundation of China under Grant 62306288, 62271452, National Key Research and Development Program of China (2022YFB4500305) and Key Research Project of Zhejiang Lab (No. 2022PI0AC01). (Corresponding author: Hongyang Chen)Linyao Yang, Hongyang Chen, Zhao Li, and Xindong Wu are with Zhejiang Lab, Hangzhou 311121, China (email: Ding is with the Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, Harbin 150001, China (email:
Abstract
Recently, ChatGPT, a representative large language model (LLM), has gained considerable attention. Due to their powerful emergent abilities, recent LLMs are considered as a possible alternative to structured knowledge bases like knowledge graphs (KGs). However, while LLMs are proficient at learning probabilistic language patterns and engaging in conversations with humans, they, like previous smaller pre-trained language models (PLMs), still have difficulty in recalling facts while generating knowledge-grounded contents. To overcome these limitations, researchers have proposed enhancing data-driven PLMs with knowledge-based KGs to incorporate explicit factual knowledge into PLMs, thus improving their performance in generating texts requiring factual knowledge and providing more informed responses to user queries. This paper reviews the studies on enhancing PLMs with KGs, detailing existing knowledge graph enhanced pre-trained language models (KGPLMs) as well as their applications. Inspired by existing studies on KGPLM, this paper proposes enhancing LLMs with KGs by developing knowledge graph-enhanced large language models (KGLLMs). KGLLM provides a solution to enhance LLMs’ factual
中文速览
大型语言模型（LLM）如ChatGPT虽然能力惊人，却始终存在一个硬伤：在需要准确事实的任务中容易"胡说八道"，因为它们只是在训练时记住了语言统计规律，而无法像知识图谱（Knowledge Graph, KG）那样显式、可靠地存储和推理事实关系。为此，这篇综述系统梳理了"用知识图谱强化预训练语言模型（KGPLM）"的研究全貌，按训练前、训练中、训练后三类增强方式对现有方法进行分类介绍，并综述了相关应用场景。在此基础上，作者进一步提出将同样的思路延伸到LLM上，构建知识图谱增强的大语言模型（KGLLM），以从根本上提升LLM的事实推理能力。这项工作不仅为研究者提供了一份全面的技术地图，也指出了LLM与KG并非替代而是互补的关系，为未来如何让LLM"知其所知、知其所不知"提供了清晰的研究方向。
原文 arXiv:2306.11489;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.11489v2