Florence: A New Foundation Model for Computer Vision
Lu Yuan Dongdong Chen Yi-Ling Chen Noel Codella Xiyang Dai Jianfeng Gao Houdong Hu Xuedong Huang Boxin Li Chunyuan Li Ce Liu Mengchen Liu Zicheng Liu Yumao Lu Yu Shi Lijuan Wang Jianfeng Wang Bin Xiao Zhen Xiao Jianwei Yang Michael Zeng Luowei Zhou Pengchuan Zhang
Abstract
Automated visual understanding of our diverse and open world demands computer vision models to generalize well with minimal customization for specific tasks, similar to human vision. Computer vision foundation models, which are trained on diverse, large-scale dataset and can be adapted to a wide range of downstream tasks, are critical for this mission to solve real-world computer vision applications. While existing vision foundation models such as CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021), and Wu Dao 2.0 (Wud, ) focus mainly on mapping images and textual representations to a cross-modal shared representation, we introduce a new computer vision foundation model, Florence, to expand the representations from coarse (scene) to fine (object), from static (images) to dynamic (videos), and from RGB to multiple modalities (caption, depth). By incorporating universal visual-language representations from Web-scale image-text data, our Florence model can be easily adapted for various computer vision tasks, such as classification, retrieval, object detection, VQA, image caption, video retrieval and action recognition. Moreover, Florence demonstrates outstanding performance in many
中文速览
大规模视觉-语言预训练模型虽已取得不错进展，但现有的 CLIP、ALIGN 等模型主要停留在图像与文本的粗粒度对齐，难以覆盖目标检测、视频理解、视觉问答等更广泛的视觉任务。微软研究院提出了视觉基础模型 Florence，通过在 9 亿图文对数据集上进行统一的图文对比学习（UniCL）预训练，并借助动态检测头、视频适配器、多模态融合适配器等轻量级模块，将模型能力从场景级延伸到目标级、从静态图像延伸到视频、从单一 RGB 延伸到字幕与深度等多种模态。Florence 在 44 个主流基准测试中的绝大多数取得了当时最优成绩，包括 ImageNet 零样本分类 top-1 准确率 83.74%、COCO 目标检测 62.4 mAP、VQA 得分 80.36 以及 Kinetics-600 动作识别 87.8% 等。这项工作的意义在于，它为计算机视觉领域提供了一个真正意义上的"通用基础模型"范式——用一套预训练骨干搭配最小化的任务适配，就能覆盖空间、时间、模态三个维度上的多样化视觉理解需求。
原文 arXiv:2111.11432;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.11432v1