A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmark
Xiaohua Zhai Joan Puigcerver Alexander Kolesnikov Pierre Ruyssen Carlos Riquelme Mario Lucic Josip Djolonga André Susano Pinto Maxim Neumann Alexey Dosovitskiy Lucas Beyer Olivier Bachem Michael Tschannen Marcin Michalski Olivier Bousquet Sylvain Gelly Neil Houlsby
Abstract
Representation learning promises to unlock deep learning for the long tail of vision tasks without expensive labelled datasets. Yet, the absence of a unified evaluation for general visual representations hinders progress. Popular protocols are often too constrained (linear classification), limited in diversity (ImageNet, CIFAR, Pascal-VOC), or only weakly related to representation quality (ELBO, reconstruction error). We present the Visual Task Adaptation Benchmark (VTAB), which defines good representations as those that adapt to diverse, unseen tasks with few examples. With VTAB, we conduct a large-scale study of many popular publicly-available representation learning algorithms. We carefully control confounders such as architecture and tuning budget. We address questions like: How effective are ImageNet representations beyond standard natural datasets? How do representations trained via generative and discriminative models compare? To what extent can self-supervision replace labels? And, how close are we to general visual representations?
中文速览
深度学习在视觉领域大放异彩，但每换一个新任务就需要大量标注数据，如何用少量样本快速适应各种任务正是表示学习（representation learning）的核心追求——而这一领域长期缺乏统一、全面的评测标准。作者提出了视觉任务适应基准（Visual Task Adaptation Benchmark，VTAB），涵盖19个横跨自然图像、专业图像（遥感、医疗）和结构理解（计数、深度估计、3D几何）三大类别的下游任务，统一用"每任务仅1000个标注样本能适应到多大程度"来衡量表示质量的好坏。基于此基准，作者对18种主流方法（有监督预训练、自监督、半监督、生成模型等）进行了大规模横向对比，严格控制了网络结构、调参预算和预训练数据等干扰因素。结果显示：有监督ImageNet预训练在自然图像任务上表现最佳，但在需要结构性理解的任务上几乎无效；自监督方法虽整体落后，却意外地在结构理解上更具优势；判别模型普遍优于生成模型；而仅用线性分类器评测会导致结论偏差。VTAB填补了表示学习跨领域统一评测的空白，为衡量"距离通用视觉表示还有多远"提供了清晰的标尺。
原文 arXiv:1910.04867;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1910.04867v2