EVA: Exploring the Limits of Masked Visual Representation Learning at Scale
Yuxin Fang2,1 Wen Wang3,1† Binhui Xie4,1† Quan Sun1 Ledell Wu1 Xinggang Wang2 Tiejun Huang1 Xinlong Wang1 Yue Cao1 1Beijing Academy of Artificial Intelligence 2Huazhong University of Science and Technology ++…3Zhejiang University 4Beijing Institute of Technology Code、Models: https://github.com/baaivision/EVA Interns at Beijing Academy of Artificial Intelligence.
Abstract
We launch EVA, a vision-centric foundation model to Explore the limits of Visual representation at scAle using only publicly accessible data. EVA is a vanilla ViT pre-trained to reconstruct the masked out image-text aligned vision features conditioned on visible image patches. Via this pretext task, we can efficiently scale up EVA to one billion parameters, and sets new records on a broad range of representative vision downstream tasks, such as image recognition, video action recognition, object detection, instance segmentation and semantic segmentation without heavy supervised training. Moreover, we observe quantitative changes in scaling EVA result in qualitative changes in transfer learning performance that are not present in other models. For instance, EVA takes a great leap in the challenging large vocabulary instance segmentation task: our model achieves almost the same state-of-the-art performance on LVISv1.0 dataset with over a thousand categories and COCO dataset with only eighty categories. Beyond a pure vision encoder, EVA can also serve as a vision-centric, multi-modal pivot to connect images and text. We find initializing the vision tower of a giant CLIP from EVA can g
中文速览
大规模视觉预训练长期依赖海量有标注数据，而纯粹的掩码图像建模（Masked Image Modeling, MIM）在十亿参数量级上又难以取得理想效果，两者之间存在明显的性能鸿沟。EVA 的做法是：用公开可得的约 3000 万张无标注图像，训练一个十亿参数的标准视觉 Transformer（ViT），让模型在遮住部分图像块后，去预测那些被遮住区域对应的 CLIP 图文对齐视觉特征——这样既保留了掩码建模对几何结构的感知能力，又引入了 CLIP 特征所携带的高层语义信息。最终，EVA 在图像分类、视频动作识别、目标检测、实例分割和语义分割等多项权威基准上全面刷新纪录，其中最引人注目的是：EVA 在拥有 1200 余个类别的 LVIS 大词表实例分割任务上，取得了与仅有 80 个类别的 COCO 数据集几乎相同的分割精度，展现出模型规模扩大后涌现出的新能力；此外，用预训练好的 EVA 初始化十亿级 CLIP 模型的视觉编码器，可以大幅稳定训练过程并以更少算力超越从零训练的基线，为多模态大模型的高效扩展提供了新思路。
原文 arXiv:2211.07636;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.07636v2