ViViT: A Video Vision Transformer
Anurag Arnab Mostafa Dehghani Georg Heigold Chen Sun Mario Lučić Cordelia Schmid Google Research {aarnab, dehghani, heigold, chensun, lucic, Equal contributionEqual advising
Abstract
We present pure-transformer based models for video classification, drawing upon the recent success of such models in image classification. Our model extracts spatio-temporal tokens from the input video, which are then encoded by a series of transformer layers. In order to handle the long sequences of tokens encountered in video, we propose several, efficient variants of our model which factorise the spatial- and temporal-dimensions of the input. Although transformer-based models are known to only be effective when large training datasets are available, we show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets. We conduct thorough ablation studies, and achieve state-of-the-art results on multiple video classification benchmarks including Kinetics 400 and 600, Epic Kitchens, Something-Something v2 and Moments in Time, outperforming prior methods based on deep 3D convolutional networks. To facilitate further research, we release code at https://github.com/google-research/scenic.
中文速览
视频理解领域长期被3D卷积网络主导，但注意力机制天然适合捕捉视频中的长程时空依赖关系，本文提出了一套名为Video Vision Transformer（ViViT）的纯Transformer视频分类模型。研究者将视频切成时空"token"块送入Transformer编码器，并针对视频序列过长导致计算量爆炸的问题，设计了多种将空间维度与时间维度解耦处理的高效变体——包括分离编码器和分解自注意力两种思路，使计算复杂度从全时空交叉注意力的平方级大幅下降。为了克服Transformer需要海量数据才能发挥效果的固有缺陷，他们系统研究了正则化策略，并充分利用预训练图像模型进行迁移，使模型在规模相对较小的视频数据集上也能稳定训练。最终，ViViT在Kinetics 400/600、Epic Kitchens、Something-Something v2和Moments in Time五个主流基准上全面超越此前最强的3D卷积网络，证明纯Transformer架构同样可以成为视频理解的有力工具。
原文 arXiv:2103.15691;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.15691v2