PRODIGY: Enabling In-context Learning Over Graphs
Qian Huang1、Hongyu Ren1∗ Peng Chen1、Gregor Kržmanc2、Daniel Zeng1 Percy Liang1、Jure Leskovec1 1Stanford University 2 University of Ljubljana indicates equal contribution.
Abstract
In-context learning is the ability of a pretrained model to adapt to novel and diverse downstream tasks by conditioning on prompt examples, without optimizing any parameters. While large language models have demonstrated this ability, how in-context learning could be performed over graphs is unexplored. In this paper, we develop Pretraining Over Diverse In-Context Graph Systems (PRODIGY), the first pretraining framework that enables in-context learning over graphs. The key idea of our framework is to formulate in-context learning over graphs with a novel prompt graph representation, which connects prompt examples and queries. We then propose a graph neural network architecture over the prompt graph and a corresponding family of in-context pretraining objectives. With PRODIGY, the pretrained model can directly perform novel downstream classification tasks on unseen graphs via in-context learning. We provide empirical evidence of the effectiveness of our framework by showcasing its strong in-context learning performance on tasks involving citation networks and knowledge graphs. Our approach outperforms the in-context learning accuracy of contrastive pretraining baselines with hard-co
中文速览
大规模预训练语言模型能够通过少量示例直接完成新任务（即上下文学习，in-context learning），但图机器学习领域一直缺乏类似能力。PRODIGY 提出了一种"提示图"（prompt graph）表示方法，把少量带标签的示例节点/边与待预测的查询节点/边以及标签节点统一连接成一张图，让模型能够在同一结构下处理节点分类、边分类等多种图任务。在此基础上，框架设计了一个两阶段图神经网络架构——先在各数据子图内做消息传递，再在任务图层面做注意力通信——并引入包括"邻域匹配"自监督目标在内的一族预训练损失，让模型在预训练阶段就学会从上下文示例中归纳规律。实验表明，PRODIGY 在从未见过的引用网络和知识图谱上，无需任何微调即可直接完成分类任务，平均比对比学习基线高出 18%，比有限数据微调高出 33%，甚至超过了在目标图上专门训练的少样本学习方法，为图学习中的上下文泛化能力提供了首个系统性解决方案。
原文 arXiv:2305.12600;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.12600v1