Combined Scaling for Zero-shot Transfer Learning
\nameHieu Pham∗ \nameZihang Dai∗ \nameGolnaz Ghiasi∗ \nameKenji Kawaguchi∗ \nameHanxiao Liu \nameAdams Wei Yu \nameJiahui Yu \nameYi-Ting Chen \nameMinh-Thang Luong \nameYonghui Wu \nameMingxing Tan \nameQuoc V. Le *: Equal contributions. Corresponding authors:
Abstract
We present a combined scaling method – named BASIC – that achieves 85.7% top-1 accuracy on the ImageNet ILSVRC-2012 validation set without learning from any labeled ImageNet example. This accuracy surpasses best-published similar models – CLIP and ALIGN – by 9.3%. Our BASIC model also shows significant improvements in robustness benchmarks. For instance, on 5 test sets with natural distribution shifts such as ImageNet-{A,R,V2,Sketch} and ObjectNet, our model achieves 84.3% top-1 average accuracy, only a small drop from its original ImageNet accuracy.
中文速览
大规模图文对比学习模型（如CLIP和ALIGN）在不使用任何标注数据的前提下虽已取得不错的图像分类性能，但与有监督方法相比仍有明显差距。研究者提出了一种名为BASIC的联合扩展方法，同时在数据规模（66亿噪声图文对，是ALIGN的4倍）、模型规模（30亿参数的CoAtNet，计算量是CLIP/ALIGN的8倍）和批量大小（65536，是ALIGN的4倍）三个维度上进行扩展，并通过梯度检查点、流水线微批处理和模型并行等工程手段解决了加速器显存不足的问题。最终，BASIC在ImageNet上实现了85.7%的零样本迁移Top-1准确率，比CLIP和ALIGN高出约9.3%，在多个自然分布偏移测试集上的平均准确率也达到84.3%，几乎没有明显下降。这项工作证明了在图文对比学习框架下，同步扩展数据、模型与批量三个维度可以显著缩小与有监督方法之间的性能差距，同时保留了零样本迁移和强鲁棒性的优势。
原文 arXiv:2111.10050;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.10050v3