Pitfalls of Graph Neural Network Evaluation
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, Stephan Günnemann Technical University of Munich, Germany Equal contribution
Abstract
Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In this paper we show that existing evaluation strategies for GNN models have serious shortcomings. We show that using the same train/validation/test splits of the same datasets, as well as making significant changes to the training procedure (e.g. early stopping criteria) precludes a fair comparison of different architectures. We perform a thorough empirical evaluation of four prominent GNN models and show that considering different splits of the data leads to dramatically different rankings of models. Even more importantly, our findings suggest that simpler GNN architectures are able to outperform the more sophisticated ones if the hyperparameters and the training procedure are tuned fairly for all models.
中文速览
图神经网络（Graph Neural Network, GNN）在半监督节点分类任务上表现亮眼，但现有的评测方式存在严重缺陷——大家总是在同一批固定划分的数据集上跑实验，还对不同模型采用各不相同的训练流程，这让"谁更好"的结论根本站不住脚。作者对四个主流GNN模型（GCN、GAT、MoNet、GraphSAGE）统一了训练流程和超参数搜索策略，并引入了多个新数据集，在每个数据集上随机生成100种数据划分、每种划分跑20次随机初始化来全面评估。结果出人意料：一旦公平比较，结构最简单的GCN反而在综合排名上胜过更复杂的模型；而换一种数据划分，各模型的排名就会发生翻天覆地的变化，说明"在一个固定划分上跑出来的分数"极不可靠。这项工作揭示了当前GNN评测体系的根本性漏洞，呼吁社区采用多次随机划分和统一训练流程来做更可信的比较，对今后如何规范深度学习基准评测具有重要参考价值。
原文 arXiv:1811.05868;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1811.05868v2