Which Modality should I use - Text, Motif, or Image? : Understanding Graphs with Large Language Models
Debarati Das Ishaan Gupta Jaideep Srivastava Dongyeop Kang Department of Computer Science, University of Minnesota {das00015, gupta737, srivasta,
Abstract
Our research integrates graph data with Large Language Models (LLMs), which, despite their advancements in various fields using large text corpora, face limitations in encoding entire graphs due to context size constraints. This paper introduces a new approach to encoding a graph with diverse modalities, such as text, image, and motif, coupled with prompts to approximate a graph’s global connectivity, thereby enhancing LLMs’ efficiency in processing complex graph structures. The study also presents GraphTMI, a novel benchmark for evaluating LLMs in graph structure analysis, focusing on homophily, motif presence, and graph difficulty. Key findings indicate that the image modality, especially with vision-language models like GPT-4V, is superior to text in balancing token limits and preserving essential information and outperforms prior graph neural net (GNN) encoders. Furthermore, the research assesses how various factors affect the performance of each encoding modality and outlines the existing challenges and potential future developments for LLMs in graph understanding and reasoning tasks. All data will be publicly available upon acceptance.
中文速览
让大型语言模型（LLM）真正"读懂"图结构数据是个难题，因为传统的纯文本编码方式在面对大型真实图时会产生冗长输入，轻易超出模型的上下文窗口限制。为此，研究者提出了三种图编码模态——文本（Text）、基序（Motif）和图像（Image）——分别用 GPT-4 和 GPT-4V 处理节点分类任务，并构建了新基准数据集 GraphTMI 来系统评估不同图难度（以同质性和基序数量衡量）下各模态的表现。实验发现，图像模态在兼顾信息完整性和 token 消耗方面优于文本模态，对中等难度任务效果最佳，而基序模态则在更难的任务中更有优势；整体来看，LLM 的图分类能力已接近传统图神经网络（GNN）基线，但仍有差距。这项工作揭示了 LLM 在图理解任务中的现实局限与潜力，为未来多模态图推理研究提供了重要参考。
原文 arXiv:2311.09862;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.09862v2