Scalable Transfer Learning with Expert Models
Joan Puigcerver Google Research、Carlos Riquelme111 Google Research、Basil Mustafa Google Research、Cedric Renggli ETH Zurich、André Susano Pinto Google Research、Sylvain Gelly Google Research、Daniel Keysers Google Research、Neil Houlsby Google Research Equal contribution. Order decided by a coin toss.Work done while interning at Google Research.
Abstract
Transfer of pre-trained representations can improve sample efficiency and reduce computational requirements for new tasks. However, representations used for transfer are usually generic, and are not tailored to a particular distribution of downstream tasks. We explore the use of expert representations for transfer with a simple, yet effective, strategy. We train a diverse set of experts by exploiting existing label structures, and use cheap-to-compute performance proxies to select the relevant expert for each target task. This strategy scales the process of transferring to new tasks, since it does not revisit the pre-training data during transfer. Accordingly, it requires little extra compute per target task, and results in a speed-up of 2–3 orders of magnitude compared to competing approaches. Further, we provide an adapter-based architecture able to compress many experts into a single model. We evaluate our approach on two different data sources and demonstrate that it outperforms baselines on over 20 diverse vision tasks in both cases.
中文速览
迁移学习中通常只用一个通用预训练模型，但它很难同时适配各种差异巨大的下游任务。这篇论文提出了一套"专家迁移"策略：先利用大规模图像数据集（如 ImageNet、JFT）自带的标签层次结构，训练出一批专注于不同子领域的"专家"模型（experts），然后为每个新任务用轻量级的性能代理指标快速挑选最合适的专家，无需再碰原始预训练数据，整个选专家过程比竞品方法快 500 到 1000 倍。为了解决存储多个完整模型代价高的问题，他们还设计了一种基于残差适配器（residual adapter）的紧凑架构，把所有专家塞进单一模型，参数量仅增加约 6% 却能保留接近完整专家的性能。在包含 19 个多样视觉任务的 VTAB 基准上，该方法比当时最优方法平均精度提升 3.6%，说明针对下游任务分布定制的专家表示，能显著提升迁移学习的效率与效果。
原文 arXiv:2009.13239;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2009.13239v1