Learning Spatio-Temporal Representation with Pseudo-3D Residual Networks
Zhaofan Qiu †, Ting Yao ‡, and Tao Mei ‡ † University of Science and Technology of China, Hefei, China ‡ Microsoft Research, Beijing, China {tiyao,
Abstract
Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for image recognition problems. Nevertheless, it is not trivial when utilizing a CNN for learning spatio-temporal video representation. A few studies have shown that performing 3D convolutions is a rewarding approach to capture both spatial and temporal dimensions in videos. However, the development of a very deep 3D CNN from scratch results in expensive computational cost and memory demand. A valid question is why not recycle off-the-shelf 2D networks for a 3D CNN. In this paper, we devise multiple variants of bottleneck building blocks in a residual learning framework by simulating $3\times 3\times 3$ convolutions with $1\times 3\times 3$ convolutional filters on spatial domain (equivalent to 2D CNN) plus $3\times 1\times 1$ convolutions to construct temporal connections on adjacent feature maps in time. Furthermore, we propose a new architecture, named Pseudo-3D Residual Net (P3D ResNet), that exploits all the variants of blocks but composes each in different placement of ResNet, following the philosophy that enhancing structural diversity with going deep could improve the power of neural networ
中文速览
视频理解领域长期面临一个两难困境：纯3D卷积神经网络虽能同时捕捉空间和时间信息，但模型庞大、训练代价极高，而基于2D CNN的帧级方法又无法充分挖掘帧间的时序关联。针对这一问题，研究者提出了"伪3D残差网络"（Pseudo-3D Residual Net，P3D ResNet），核心思路是把一个3×3×3的3D卷积拆解为一个处理空间信息的1×3×3卷积（可直接复用预训练2D CNN权重）和一个处理时间信息的3×1×1卷积，并在残差学习框架下设计了串联、并联、混合三种不同连接方式的模块（P3D-A/B/C），再将这三种模块交替搭建成一个深层网络以增强结构多样性。实验结果显示，P3D ResNet在Sports-1M大规模视频分类任务上分别比3D CNN和帧级2D CNN高出5.3%和1.8%，在另外五个基准数据集的三类下游任务中也全面超越当时的主流方法。这项工作的重要意义在于，它以极低的额外计算代价实现了接近甚至超越真正3D卷积的时空建模能力，并提供了一个可复用图像预训练模型的通用视频表征学习框架。
原文 arXiv:1711.10305;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1711.10305v1