Beyond Short Snippets: Deep Networks for Video Classification
Joe Yue-Hei Ng1 Matthew Hausknecht2 Sudheendra Vijayanarasimhan3 Oriol Vinyals3 Rajat Monga3 George Toderici3 1University of Maryland, College Park 2University of Texas at Austin 3Google, Inc.
Abstract
Convolutional neural networks (CNNs) have been extensively applied for image recognition problems giving state-of-the-art results on recognition, detection, segmentation and retrieval. In this work we propose and evaluate several deep neural network architectures to combine image information across a video over longer time periods than previously attempted. We propose two methods capable of handling full length videos. The first method explores various convolutional temporal feature pooling architectures, examining the various design choices which need to be made when adapting a CNN for this task. The second proposed method explicitly models the video as an ordered sequence of frames. For this purpose we employ a recurrent neural network that uses Long Short-Term Memory (LSTM) cells which are connected to the output of the underlying CNN. Our best networks exhibit significant performance improvements over previously published results on the Sports 1 million dataset (73.1% vs. 60.9%) and the UCF-101 datasets with (88.6% vs. 88.0%) and without additional optical flow information (82.6% vs. 73.0%).
中文速览
视频动作识别的难点在于如何把散落在数百乃至数千帧中的信息整合成一个全局描述，而此前的卷积神经网络（CNN）方案大多只处理几秒钟的短片段，难以捕捉长时序规律。本文提出两类解决方案：一是在CNN不同层级插入时序最大池化（temporal max-pooling）来聚合全视频的帧级特征；二是将CNN提取的逐帧特征输入长短期记忆网络（LSTM），让模型显式地学习帧间的时序依赖关系。为兼顾运动信息与计算效率，作者以每秒一帧的低帧率采样，再单独引入光流图像来补偿丢失的运动线索，从而在保持参数量不随视频长度增长的前提下获得全局描述。实验结果显示，所提方法在Sports-1M数据集上将准确率从60.9%提升至73.1%，在UCF-101上也刷新了当时的最优成绩，证明将长时序建模与显式运动信息相结合是提升视频分类性能的有效路径。
原文 arXiv:1503.08909;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1503.08909v2