Elucidating image-to-set prediction: An analysis of models, losses and datasets
Luis Pineda1, Amaia Salvador , Michal Drozdzal1, Adriana Romero1 1Facebook AI Research 2Universitat Politecnica de Catalunya {lep, mdrozdzal, Equal contribution.Work partially done during internship at Facebook AI Research.
Abstract
In this paper, we identify an important reproducibility challenge in the image-to-set prediction literature that impedes proper comparisons among published methods, namely, researchers use different evaluation protocols to assess their contributions. To alleviate this issue, we introduce an image-to-set prediction benchmark suite built on top of five public datasets of increasing task complexity that are suitable for multi-label classification (VOC, COCO, NUS-WIDE, ADE20k and Recipe1M). Using the benchmark, we provide an in-depth analysis where we study the key components of current models, namely the choice of the image representation backbone as well as the set predictor design. Our results show that (1) exploiting better image representation backbones leads to higher performance boosts than enhancing set predictors, and (2) modeling both the label co-occurrences and ordering has a slight positive impact in terms of performance, whereas explicit cardinality prediction only helps when training on complex datasets, such as Recipe1M. To facilitate future image-to-set prediction research, we make the code, best models and dataset splits publicly available at: https://github.com/faceb
中文速览
多标签分类（image-to-set prediction）领域长期存在一个被忽视的可复现性危机：不同论文使用不同的数据划分、评估指标和实验设置，导致方法之间根本无法公平比较。为此，作者在五个难度递增的公开数据集（VOC、COCO、NUS-WIDE、ADE20k、Recipe1M）上构建了一套统一的基准测试框架，并用固定超参数搜索预算对主流模型的核心组件——图像特征主干网络与集合预测模块——进行了系统性分析。结果发现，换用更强的图像主干网络（如从 ResNet-50 升级到 ResNeXt-101）带来的性能提升远大于精心设计集合预测模块的收益；建模标签共现关系和预测顺序能带来小幅改善，而显式预测集合大小（cardinality prediction）只在 Recipe1M 这类复杂数据集上才有明显帮助。这项工作的意义在于：它不仅揭示了该领域结论不可靠的根源，还提供了可直接复用的代码、模型和数据划分，为后续研究奠定了公平比较的基础。
原文 arXiv:1904.05709;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1904.05709v2