Beyond Text: A Deep Dive into Large Language Models’ Ability on Understanding Graph Data
Yuntong Hu∗, Zheng Zhang , Liang Zhao Department of Computer Science Emory University Atlanta, GA 30322, USA These authors contributed equally to this work.
Abstract
Large language models (LLMs) have achieved impressive performance on many natural language processing tasks. However, their capabilities on graph-structured data remain relatively unexplored. In this paper, we conduct a series of experiments benchmarking leading LLMs on diverse graph prediction tasks spanning node, edge, and graph levels. We aim to assess whether LLMs can effectively process graph data and leverage topological structures to enhance performance, compared to specialized graph neural networks. Through varied prompt formatting and task/dataset selection, we analyze how well LLMs can interpret and utilize graph structures. By comparing LLMs’ performance with specialized graph models, we offer insights into the strengths and limitations of employing LLMs for graph analytics. Our findings provide insights into LLMs’ capabilities and suggest avenues for further exploration in applying them to graph analytics.
中文速览
大语言模型（LLMs）在文本任务上大放异彩，但它们能不能处理图结构数据（graph-structured data）却少有人系统研究。这篇论文用GPT-3.5和GPT-4在节点分类、链路预测、图分类等多个层级的任务上做了全面测评，并通过设计不同的提示词（prompt）来考察模型能否理解和利用图的拓扑结构。结果显示：在节点分类上LLMs通常弱于专门训练的图神经网络（GNN），但在链路预测上，一旦把邻居信息加入提示词，GPT-4的零样本表现甚至超越了所有有监督GNN基线；不过，结构信息并非万能，过多细节反而会拖累预测精度，且不同任务、不同数据集领域对结构信息的依赖程度差异显著。这项研究首次系统摸清了LLMs在图分析中的能力边界，为后续将大模型引入图学习领域指出了有潜力的方向，也点明了当前的关键瓶颈。
原文 arXiv:2310.04944;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.04944v1