Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction
Eli Chien University of Illinois Urbana-Champaign, USA、Wei-Cheng Chang Amazon, USA \ANDCho-Jui Hsieh University of California, Los Angeles, USA、Hsiang-Fu Yu, Jiong Zhang Amazon, USA \ANDOlgica Milenkovic University of Illinois Urbana-Champaign, USA、Inderjit S. Dhillon Amazon, USA This work was done during Eli Chien’s internship at Amazon, USA.
Abstract
Learning on graphs has attracted significant attention in the learning community due to numerous real-world applications. In particular, graph neural networks (GNNs), which take numerical node features and graph structure as inputs, have been shown to achieve state-of-the-art performance on various graph-related learning tasks. Recent works exploring the correlation between numerical node features and graph structure via self-supervised learning have paved the way for further performance improvements of GNNs. However, methods used for extracting numerical node features from raw data are still graph-agnostic within standard GNN pipelines. This practice is sub-optimal as it prevents one from fully utilizing potential correlations between graph topology and node attributes. To mitigate this issue, we propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT). GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information, and scales to large datasets. We also provide a theoretical analysis that justifies the use of XMC over link prediction and motiv
中文速览
大规模图上的节点通常携带文本信息（如论文标题、商品描述），现有方法用词袋或预训练BERT将文本转成数值特征后再喂给图神经网络（GNN），整个特征提取过程完全忽略了图的拓扑结构，导致性能受限。为此，作者提出GIANT框架，核心思路是让语言模型（BERT）以"邻居预测"为自监督任务接受微调——把每个节点的邻居集合编码为多标签，要求模型仅凭节点文本预测出图中哪些节点与其相邻，从而将图结构信息注入特征提取过程；为了应对节点数达百万级时标签空间极大的挑战，他们将该任务形式化为超大规模多标签分类（eXtreme Multi-label Classification，XMC）问题，并引入XR-Transformer这一高效分层求解器来完成训练。在Open Graph Benchmark的大规模数据集上，GIANT生成的节点特征可为各类下游GNN带来显著提升，例如在ogbn-papers100M上将MLP精度从47.24%提升至61.10%、将排名第一的GAMLP从68.25%提升至69.67%，证明在特征提取阶段引入图拓扑监督是一条简单有效且普适的改进路径。
原文 arXiv:2111.00064;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.00064v3