HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention
Shijie Geng1,2 , Jianbo Yuan2, Yu Tian2, Yuxiao Chen1, Yongfeng Zhang1 1Rutgers University 2ByteDance Inc. {sg1309, yc984, {jianbo.yuan, This work was conducted while interning at ByteDance.
Abstract
The success of large-scale contrastive vision-language pretraining (CLIP) has benefited both visual recognition and multimodal content understanding. The concise design brings CLIP the advantage in inference efficiency against other vision-language models with heavier cross-attention fusion layers, making it a popular choice for a wide spectrum of downstream tasks. However, CLIP does not explicitly capture the hierarchical nature of high-level and fine-grained semantics conveyed in images and texts, which is arguably critical to vision-language understanding and reasoning. To this end, we equip both the visual and language branches in CLIP with hierarchy-aware attentions, namely Hierarchy-aware CLIP (HiCLIP), to progressively discover semantic hierarchies layer-by-layer from both images and texts in an unsupervised manner. As a result, such hierarchical aggregation significantly improves the cross-modal alignment. To demonstrate the advantages of HiCLIP, we conduct qualitative analysis on its unsupervised hierarchy induction during inference, as well as extensive quantitative experiments on both visual recognition and vision-language downstream tasks.111We release our implementatio
中文速览
大规模对比式视觉-语言预训练模型CLIP虽然推理高效、泛化能力强，却无法显式捕捉图像和文本中从局部到整体的层级语义结构（semantic hierarchy），而这种结构对深层的跨模态理解与推理至关重要。为此，研究者提出HiCLIP，在CLIP的视觉编码器和语言编码器中分别引入"层级感知注意力"（hierarchy-aware attention）：通过计算相邻图像块或文本词元之间的亲和度分数，并在Transformer逐层加深的过程中以不可拆分的方式累积传播，使模型无需任何标注即可自底向上地自动发现视觉对象群组和文本短语结构。在大规模图文数据上预训练后，HiCLIP在视觉识别、图文检索、视觉问答和视觉蕴含推理等多项下游任务上均优于原版CLIP及同类方法，同时还能可视化图像与文本的层级演化过程，提升了模型的可解释性。这项工作表明，将人类感知世界的层级结构作为归纳偏置嵌入对比学习框架，是一种低成本却有效提升多模态表征质量的新思路。
原文 arXiv:2303.02995;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.02995v1