Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos
Rui Hou, Chen Chen, Mubarak Shah Center for Research in Computer Vision (CRCV), University of Central Florida (UCF)
Abstract
Deep learning has been demonstrated to achieve excellent results for image classification and object detection. However, the impact of deep learning on video analysis (e.g. action detection and recognition) has been limited due to complexity of video data and lack of annotations. Previous convolutional neural networks (CNN) based video action detection approaches usually consist of two major steps: frame-level action proposal generation and association of proposals across frames. Also, most of these methods employ two-stream CNN framework to handle spatial and temporal feature separately. In this paper, we propose an end-to-end deep network called Tube Convolutional Neural Network (T-CNN) for action detection in videos. The proposed architecture is a unified deep network that is able to recognize and localize action based on 3D convolution features. A video is first divided into equal length clips and next for each clip a set of tube proposals are generated based on 3D Convolutional Network (ConvNet) features. Finally, the tube proposals of different clips are linked together employing network flow and spatio-temporal action detection is performed using these linked video proposals
中文速览
视频动作检测（action detection）不仅要识别视频里发生了什么动作，还要精确定位动作在哪段时间、画面哪个区域出现，难点在于视频数据量大、时空搜索空间复杂，且现有方法大多先逐帧生成候选框再跨帧拼接，空间和时间信息被割裂处理。本文提出了一个叫做"管状卷积神经网络"（Tube CNN，T-CNN）的端到端深度学习框架：把视频切成等长片段，用三维卷积网络（3D ConvNet）同时抓取空间和时间特征，再通过新设计的"管状候选网络"（TPN）生成三维时空候选管道，然后借助网络流算法将各片段的候选管道串联成完整的时空检测结果，并引入新的"感兴趣管道池化"（ToI Pooling）层来处理不同尺寸的候选区域。在UCF-Sports、J-HMDB、UCF-101和THUMOS'14等多个主流数据集上的实验表明，T-CNN在修剪和未修剪视频中均达到了当时最优的动作检测性能，意义在于首次将3D卷积统一应用于动作检测的候选生成与识别全流程，省去了光流单独计算等繁琐步骤。
原文 arXiv:1703.10664;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1703.10664v3