Video ChatCaptioner: Towards Enriched Spatiotemporal Descriptions
Jun Chen Deyao Zhu Kilichbek Haydarov Xiang Li Mohamed Elhoseiny King Abdullah University of Science and Technology {jun.chen, deyao.zhu, kilichbek.haydarov, xiang.li.1,
Abstract
Video captioning aims to convey dynamic scenes from videos using natural language, facilitating the understanding of spatiotemporal information within our environment. Although there have been recent advances, generating detailed and enriched video descriptions continues to be a substantial challenge. In this work, we introduce Video ChatCaptioner, an approach for creating more comprehensive spatiotemporal video descriptions. Our method employs a ChatGPT model as a controller, specifically designed to select frames for posing video content-driven questions. Subsequently, BLIP-2 is utilized to answer these visual queries. This question-answer framework effectively uncovers intricate video details and shows promise as a method for enhancing video content. Following multiple conversational rounds, ChatGPT can summarize enriched video content based on previous conversations. Through the human evaluation experiments, we found that nearly 62.5% of participants agree that Video ChatCaptioner can cover more visual information compared to ground-truth captions. The code is publicly available at https://github.com/Vision-CAIR/ChatCaptioner
中文速览
视频字幕生成长期面临描述内容过于简单的问题——现有数据集的标注往往只覆盖基本动作，忽略背景、细节和时序关系。为此，研究者提出了 Video ChatCaptioner，让 ChatGPT 充当"提问控制器"，针对视频中均匀采样的关键帧自动生成有针对性的视觉问题，再由图文模型 BLIP-2 逐帧作答，经过多轮问答对话后由 ChatGPT 汇总生成完整、丰富的视频描述。这一方法完全绕开了对视频-文本配对数据集的依赖，巧妙地将图像理解能力扩展到时空维度。在 MSVD 和 WebViD 数据集上的人工评估表明，62.5% 的参与者认为该方法生成的字幕比人工标注的真实字幕覆盖了更多准确的视觉信息，为无需专门视频预训练即可生成精细视频描述提供了一条新思路。
原文 arXiv:2304.04227;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2304.04227v3