Exploiting Image-trained CNN Architectures for Unconstrained Video Classification
Shengxin Zha Northwestern University Evanston IL USA Florian Luisier, Walter Andrews Raytheon BBN Technologies Cambridge, MA USA Nitish Srivastava, Ruslan Salakhutdinov University of Toronto
Abstract
We conduct an in-depth exploration of different strategies for doing event detection in videos using convolutional neural networks (CNNs) trained for image classification. We study different ways of performing spatial and temporal pooling, feature normalization, choice of CNN layers as well as choice of classifiers. Making judicious choices along these dimensions led to a very significant increase in performance over more naive approaches that have been used till now. We evaluate our approach on the challenging TRECVID MED’14 dataset with two popular CNN architectures pretrained on ImageNet. On this MED’14 dataset, our methods, based entirely on image-trained CNN features, can outperform several state-of-the-art non-CNN models. Our proposed late fusion of CNN- and motion-based features can further increase the mean average precision (mAP) on MED’14 from $34.95\%$ to $38.74\%$ . The fusion approach achieves the state-of-the-art classification performance on the challenging UCF-101 dataset.
中文速览
视频中的事件检测（event detection）长期依赖手工设计的时空特征，但如何把仅在图像上训练好的卷积神经网络（CNN）高效迁移到视频任务上，始终缺乏系统性研究。研究者对这一问题展开了全面探索，从CNN层的选择、帧采样方式、空间与时间池化策略、特征归一化方法到分类器选型，逐一进行对比实验，找出最优组合。在极具挑战性的TRECVID MED'14大规模视频数据集上，仅靠图像训练的CNN特征就已超越多种依赖手工运动特征的先进方法；进一步将CNN特征与运动轨迹特征（improved dense trajectories）做后期融合后，mAP从34.95%提升至38.74%，同时在UCF-101动作识别数据集上也达到了当时最优水平。这项工作的重要性在于，它为后续专门针对视频设计的时空CNN模型提供了一条坚实的二维CNN基准线，证明充分挖掘图像预训练模型的潜力，其收益不亚于从头训练复杂的视频专用网络。
原文 arXiv:1503.04144;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1503.04144v3