LLM4DyG: Can Large Language Models Solve Spatial-Temporal Problems on Dynamic Graphs?Conference: Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August 25–29, 2024; Barcelona, SpainProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’24), August 25–29, 2024, Barcelona, SpainDOI: 10.1145/3637528.3671709ISBN: 979-8-4007-0490-1/24/08CCS: Information systems Data miningCCS: Computing methodologies Natural language processingCCS: Computing methodologies Knowledge representation and reasoning
Zeyang Zhang OrcID: 0000-0003-1329-1313 Affiliation: DCST, Tsinghua University , Beijing , China email: , Xin Wang OrcID: 0000-0002-0351-2939 Affiliation: DCST, BNRist, Tsinghua University , Beijing , China email: , Ziwei Zhang OrcID: 0000-0003-2451-843X Affiliation: DCST, Tsinghua University , Beijing , China email: , Haoyang Li OrcID: 0000-0003-3544-5563 Affiliation: DCST, Tsinghua University , Beijing , China email: , Yijian Qin OrcID: 0000-0002-0419-5226 Affiliation: DCST, Tsinghua University , Beijing , China email: and Wenwu Zhu Note: Corresponding Authors. OrcID: 0000-0003-2236-9290 Affiliation: DCST, BNRist, Tsinghua University , Beijing , China email:
Abstract
In an era marked by the increasing adoption of Large Language Models (LLMs) for various tasks, there is a growing focus on exploring LLMs’ capabilities in handling web data, particularly graph data. Dynamic graphs, which capture temporal network evolution patterns, are ubiquitous in real-world web data. Evaluating LLMs’ competence in understanding spatial-temporal information on dynamic graphs is essential for their adoption in web applications, which remains unexplored in the literature. In this paper, we bridge the gap via proposing to evaluate LLMs’ spatial-temporal understanding abilities on dynamic graphs, to the best of our knowledge, for the first time. Specifically, we propose the LLM4DyG benchmark, which includes nine specially designed tasks considering the capability evaluation of LLMs from both temporal and spatial dimensions. Then, we conduct extensive experiments to analyze the impacts of different data generators, data statistics, prompting techniques, and LLMs on the model performance. Finally, we propose Disentangled Spatial-Temporal Thoughts (DST2) for LLMs on dynamic graphs to enhance LLMs’ spatial-temporal understanding abilities. Our main observations are: 1) L
原文 arXiv:2310.17110;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.17110v3