Learning on Large-scale Text-attributed Graphs via Variational Inference
Jianan Zhao1,3 , Meng Qu1,3∗, Chaozhuo Li2†, Hao Yan4, Qian Liu5, Rui Li6, Xing Xie2, Jian Tang1,7,8† 1Mila - Québec AI Institute, 2Microsoft Research Asia, 3Université de Montréal 4Central South University, 5Sea AI Lab, 6Dalian University of Technology 7HEC Montréal, 8Canadian Institute for Advanced Research (CIFAR) The first two authors contributed equally. †Corresponding authors.
Abstract
This paper studies learning on text-attributed graphs (TAGs), where each node is associated with a text description. An ideal solution for such a problem would be integrating both the text and graph structure information with large language models and graph neural networks (GNNs). However, the problem becomes very challenging when graphs are large due to the high computational complexity brought by training large language models and GNNs together. In this paper, we propose an efficient and effective solution to learning on large text-attributed graphs by fusing graph structure and language learning with a variational Expectation-Maximization (EM) framework, called GLEM. Instead of simultaneously training large language models and GNNs on big graphs, GLEM proposes to alternatively update the two modules in the E-step and M-step. Such a procedure allows training the two modules separately while simultaneously allowing the two modules to interact and mutually enhance each other. Extensive experiments on multiple data sets demonstrate the efficiency and effectiveness of the proposed approach 111Codes are available at https://github.com/AndyJZhao/GLEM..
中文速览
文本属性图（text-attributed graph，TAG）中每个节点都带有文字描述，如何同时利用文本语义和图结构来学习节点表示，是一个很有价值但计算代价极高的问题——直接把大语言模型（LM）和图神经网络（GNN）拼在一起训练，在大规模图上内存会爆炸。为此，论文提出了 GLEM 方法，借助变分期望最大化（variational EM）框架，让 LM 和 GNN 交替更新而非同步训练：E 步固定 GNN、用它产生的伪标签来优化 LM，M 步固定 LM、用它输出的节点表示和伪标签来优化 GNN，两个模块互相"教"对方，把局部文本信息和全局结构信息融合进各自的参数里。在 ogbn-arxiv、ogbn-products、ogbn-papers100M 三个大规模基准数据集上，GLEM 的 GNN 端刷新了当时的最优结果，LM 端也达到甚至超过了现有 GNN 模型的水平，同时大幅降低了显存占用。这项工作的意义在于，它给出了一条真正可扩展、又不牺牲性能的 LM 与 GNN 联合建模路径，对社交网络、学术引用图等大规模实际场景有直接的应用价值。
原文 arXiv:2210.14709;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2210.14709v2