SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning
Keyu Duan1,2*, Qian Liu2, Tat-Seng Chua1, Shuicheng Yan2, Wei Tsang Ooi1†, Qizhe Xie1†, Junxian He3† 1 National University of Singapore 2 Sea AI Lab, Singapore 3 The Hong Kong University of Science and Technology *Correspondence to Keyu Duan advising
Abstract
Textual graphs (TGs) are graphs whose nodes correspond to text (sentences or documents), which are widely prevalent. The representation learning of TGs involves two stages: $(i)$ unsupervised feature extraction and $(ii)$ supervised graph representation learning. In recent years, extensive efforts have been devoted to the latter stage, where Graph Neural Networks (GNNs) have dominated. However, the former stage for most existing graph benchmarks still relies on traditional feature engineering techniques. More recently, with the rapid development of language models (LMs), researchers have focused on leveraging LMs to facilitate the learning of TGs, either by jointly training them in a computationally intensive framework (merging the two stages), or designing complex self-supervised training tasks for feature extraction (enhancing the first stage). In this work, we present SimTeG, a frustratingly Simple approach for Textual Graph learning that does not innovate in frameworks, models, and tasks. Instead, we first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task, such as node classification. We then generate node embeddings using the
中文速览
文本图（Textual Graph）上的节点分类和链接预测长期依赖词袋、Skip-gram等老旧特征，而将语言模型（LM）引入图学习的已有方法又普遍设计繁琐、训练代价高。SimTeG提出了一条出人意料地简单的两阶段流程：先用参数高效微调（PEFT）在下游任务标签的监督下微调预训练语言模型，再用微调后的LM对节点文本做均值池化生成嵌入，最后把这些嵌入直接喂给任意GNN完成训练。实验表明，这一做法在节点分类和链接预测多个基准上均大幅超越原始特征及未微调LM特征，一个普通的两层GraphSAGE搭配SimTeG即可在OGBN-Arxiv上达到77.48%，结合最优GNN更推进到78.04%的新最优水平。这项工作说明，图学习性能的瓶颈很大程度上卡在特征质量而非GNN结构复杂度上，好的文本特征提取本身就足以带来显著收益，为后续研究提供了一条简洁而强力的基线。
原文 arXiv:2308.02565;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.02565v1