Initialization Strategies of Spatio-Temporal Convolutional Neural Networks
Elman Mansimov Department of Computer Science, University of Toronto Nitish Srivastava and Ruslan Salakhutdinov emansim, nitish, Department of Computer Science, University of Toronto
Abstract
We propose a new way of incorporating temporal information present in videos into Spatial Convolutional Neural Networks (ConvNets) trained on images, that avoids training Spatio-Temporal ConvNets from scratch. We describe several initializations of weights in 3D Convolutional Layers of Spatio-Temporal ConvNet using 2D Convolutional Weights learned from ImageNet. We show that it is important to initialize 3D Convolutional Weights judiciously in order to learn temporal representations of videos. We evaluate our methods on the UCF-101 dataset and demonstrate improvement over Spatial ConvNets.
中文速览
如何让只在图片上训练过的卷积神经网络（Spatial ConvNet）也能"看懂"视频里的时序动作，是视频动作识别的核心难题。研究者提出了四种将ImageNet预训练的2D卷积权重迁移到3D时空卷积层（Spatio-Temporal ConvNet）的初始化方法，核心思路是让权重之和保持与原2D权重一致，从而避免从零训练耗时且容易过拟合的问题。实验发现，"零权重初始化"和"负权重初始化"效果最好——因为各子矩阵之间差异越大，网络越容易跳出只提取空间特征的局部最优，进而学到跨帧的时序表示；而简单平均或缩放初始化则因各子矩阵过于相似，网络几乎学不到额外的时序信息。最终，该方法结合复合LSTM在UCF-101数据集上达到85.3%的动作识别准确率，不仅超越了在百万级大规模视频数据集上从头训练的时空卷积网络，也为在小规模视频数据上高效构建时空模型提供了切实可行的思路。
原文 arXiv:1503.07274;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1503.07274v1