Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
Antoine Yang1,2, Antoine Miech3, Josef Sivic4, Ivan Laptev1,2, Cordelia Schmid1,2 1Inria Paris 2Département d’informatique de l’ENS, CNRS, PSL Research University 3DeepMind 4CIIRC CTU Prague https://antoyang.github.io/frozenbilm.html
Abstract
Video question answering (VideoQA) is a complex task that requires diverse multi-modal data for training. Manual annotation of question and answers for videos, however, is tedious and prohibits scalability. To tackle this problem, recent methods consider zero-shot settings with no manual annotation of visual question-answer. In particular, a promising approach adapts frozen autoregressive language models pretrained on Web-scale text-only data to multi-modal inputs. In contrast, we here build on frozen bidirectional language models (BiLM) and show that such an approach provides a stronger and cheaper alternative for zero-shot VideoQA. In particular, (i) we combine visual inputs with the frozen BiLM using light trainable modules, (ii) we train such modules using Web-scraped multi-modal data, and finally (iii) we perform zero-shot VideoQA inference through masked language modeling, where the masked text is the answer to a given question. Our proposed approach, FrozenBiLM, outperforms the state of the art in zero-shot VideoQA by a significant margin on a variety of datasets, including LSMDC-FiB, iVQA, MSRVTT-QA, MSVD-QA, ActivityNet-QA, TGIF-FrameQA, How2QA and TVQA. It also demonstrat
中文速览
人工标注视频问答数据既昂贵又难以大规模获取，如何让模型在完全没有视频问答标注的情况下回答视频中的问题，是这篇论文想解决的核心难题。作者提出了 FrozenBiLM 框架：将权重冻结的双向语言模型（Bidirectional Language Model, BiLM，如 BERT 类模型）与冻结的视觉编码器通过轻量级适配器（adapter）模块相连，再用网络上自动爬取的视频-文字配对数据训练这些小模块，最终把视频问答转化为"完形填空"形式，让双向语言模型用掩码语言建模直接预测答案。在八个主流零样本视频问答基准上，FrozenBiLM 大幅超越此前依赖大规模自回归语言模型的方法，同时训练成本更低；在少样本和全监督设置下也展现出有竞争力的性能。这项工作表明，双向语言模型是构建低成本、免标注视频理解系统的更优选择，对推动多模态理解研究具有重要价值。
原文 arXiv:2206.08155;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.08155v2