Prompting Visual-Language Models for Efficient Video Understanding
Chen Ju 11 Tengda Han 22 Kunhao Zheng 11 Ya Zhang 11 Weidi Xie 1122
Abstract
Image-based visual-language (I-VL) pre-training has shown great success for learning joint visual-textual representations from large-scale web data, revealing remarkable ability for “zero-shot” generalisation. This paper presents a simple but strong baseline to efficiently adapt the pre-trained I-VL model, and exploit its powerful ability for resource-hungry video understanding tasks, with minimal training. Specifically, we propose to optimise a few random vectors, termed as “continuous prompt vectors”, that convert video-related tasks into the same format as the pre-training objectives. In addition, to bridge the gap between static images and videos, temporal information is encoded with lightweight Transformers stacking on top of frame-wise visual features. Experimentally, we conduct extensive ablation studies to analyse the critical components. On $10$ public benchmarks of action recognition, action localisation, and text-video retrieval, across closed-set, few-shot, and zero-shot scenarios, we achieve competitive or state-of-the-art performance to existing methods, despite optimising significantly fewer parameters.
中文速览
图像-文本预训练模型（如CLIP）在零样本图像分类上表现出色，但如何高效地把它迁移到视频理解这类"数据饥渴"型任务上仍是一个开放问题。研究者提出了一种简洁而有效的方案：冻结预训练的CLIP参数，只优化少量随机初始化的"连续提示向量"（continuous prompt vectors），将动作识别、动作定位、文本-视频检索等视频任务统一转化为与预训练目标相同的视觉-文本匹配形式；同时在帧级视觉特征之上叠加轻量级Transformer来捕捉时序信息。在10个公开基准上，覆盖闭集、少样本和零样本三种场景的实验表明，该方法在只训练极少量参数（几个提示向量加两层Transformer）的情况下，就达到了与现有方法相当甚至更优的性能，在7个少样本/零样本基准上有时领先超过10个百分点。这项工作的意义在于，它为社区提供了一条将强大的图像-语言基础模型低成本迁移至视频领域的清晰路径，且天然保留了模型的零样本泛化能力。
原文 arXiv:2112.04478;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.04478v2