Efficient Video Classification Using Fewer Frames
Shweta Bhardwaj Indian Institute of Technology Madras and Robert Bosch Centre for Data Science and AI (RBC-DSAI) Mukundhan Srinivasan NVIDIA Bangalore Mitesh M. Khapra††
Abstract
Recently, there has been a lot of interest in building compact models for video classification which have a small memory footprint ( $<$ 1 GB) [15]. While these models are compact, they typically operate by repeated application of a small weight matrix to all the frames in a video. For example, recurrent neural network based methods compute a hidden state for every frame of the video using a recurrent weight matrix. Similarly, cluster-and-aggregate based methods such as NetVLAD have a learnable clustering matrix which is used to assign soft-clusters to every frame in the video. Since these models look at every frame in the video, the number of floating point operations (FLOPs) is still large even though the memory footprint is small. In this work, we focus on building compute-efficient video classification models which process fewer frames and hence have less number of FLOPs. Similar to memory efficient models, we use the idea of distillation albeit in a different setting. Specifically, in our case, a compute-heavy teacher which looks at all the frames in the video is used to train a compute-efficient student which looks at only a small fraction of frames in the video. This is in c
中文速览
视频分类模型普遍存在一个矛盾：即便把模型压缩得很小、省内存，它仍然要逐帧处理整段视频，导致计算量（FLOPs）依然巨大。为解决这一问题，研究者提出用"知识蒸馏（knowledge distillation）"的思路训练一对师生网络——教师模型看完视频所有帧，学生模型只看其中一小部分帧，通过让学生模仿教师的中间表示和输出分布来弥补信息损失。他们在循环神经网络（H-RNN）、NetVLAD 和内存高效的 NeXtVLAD 三类主流视频分类架构上验证了这一方法，实验基于 YouTube-8M 大规模数据集。结果表明，学生网络可将推理时间缩短约 30%、将浮点计算量降低约 90%，而分类性能几乎没有明显下降。这项工作填补了现有研究只关注"省内存"而忽视"省计算"的空白，为在手机等低功耗设备上实时处理长视频提供了切实可行的解决路径。
原文 arXiv:1902.10640;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1902.10640v1