Large-Scale Representation Learning on Graphs via Bootstrapping
Shantanu Thakoor DeepMind、Corentin Tallec DeepMind、Mohammad Gheshlaghi Azar DeepMind、Mehdi Azabou Georgia Institute of Technology、Eva Dyer Georgia Institute of Technology、Rémi Munos DeepMind、Petar Veličković DeepMind、Michal Valko DeepMind Correspondence to: Shantanu Thakoor
Abstract
Self-supervised learning provides a promising path towards eliminating the need for costly label information in representation learning on graphs. However, to achieve state-of-the-art performance, methods often need large numbers of negative examples and rely on complex augmentations. This can be prohibitively expensive, especially for large graphs. To address these challenges, we introduce Bootstrapped Graph Latents (BGRL) - a graph representation learning method that learns by predicting alternative augmentations of the input. BGRL uses only simple augmentations and alleviates the need for contrasting with negative examples, and is thus scalable by design. BGRL outperforms or matches prior methods on several established benchmarks, while achieving a 2-10x reduction in memory costs. Furthermore, we show that BGRL can be scaled up to extremely large graphs with hundreds of millions of nodes in the semi-supervised regime - achieving state-of-the-art performance and improving over supervised baselines where representations are shaped only through label information. In particular, our solution centered on BGRL constituted one of the winning entries to the Open Graph Benchmark - Large
中文速览
图神经网络的自监督表示学习一直依赖对比学习，需要大量负样本（negative examples）做"排斥"操作，导致计算和内存开销随节点数平方级增长，难以应用于超大规模图。本文提出了 BGRL（Bootstrapped Graph Latents），借鉴计算机视觉中 BYOL 的自举（bootstrapping）思路，用一个在线编码器预测目标编码器的表示，目标编码器则以指数移动平均方式更新，整个流程完全不需要负样本，复杂度随图规模线性增长。在多个中小规模标准基准上，BGRL 的表现与最优对比学习方法持平甚至更好，同时内存占用降低 2–10 倍；更重要的是，BGRL 成功扩展到拥有 2.4 亿节点、17 亿条边的超大规模图 MAG240M，凭借半监督训练达到当时最优性能，并摘得 KDD Cup 2021 大规模图挑战赛的冠军，证明了无需负样本的自监督学习在工业级超大图上的实用价值。
原文 arXiv:2102.06514;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.06514v3