Temporal Modeling Approaches for Large-scale Youtube-8M Video Understanding
Fu Li, Chuang Gan, Xiao Liu, Yunlong Bian, Xiang Long, Yandong Li, Zhichao Li, Jie Zhou, Shilei Wen Baidu IDL、Tsinghua University
Abstract
This paper describes our solution for the video recognition task of the Google Cloud & YouTube-8M Video Understanding Challenge that ranked the 3rd place. Because the challenge provides pre-extracted visual and audio features instead of the raw videos, we mainly investigate various temporal modeling approaches to aggregate the frame-level features for multi-label video recognition. Our system contains three major components: two-stream sequence model, fast-forward sequence model and temporal residual neural networks. Experiment results on the challenging Youtube-8M dataset demonstrate that our proposed temporal modeling approaches can significantly improve existing temporal modeling approaches in the large-scale video recognition tasks. To be noted, our fast-forward LSTM with a depth of 7 layers achieves 82.75% in term of GAP@20 on the Kaggle Public test set.
中文速览
大规模视频理解竞赛中，因为主办方只提供预提取的视觉和音频特征而非原始视频，如何有效整合逐帧特征、捕捉视频时序关系成为核心难题。针对这一挑战，研究团队提出了三种时序建模方法：双流序列模型（用两个独立的双向LSTM/GRU分别处理视觉和音频特征再融合）、快速前向序列模型（通过在多层LSTM间添加快速前向连接解决深层循环网络难以优化的问题，最终成功训练出7层深度模型）、以及时序残差卷积网络（先用带残差连接的时序卷积提取更具判别性的特征序列，再送入LSTM分类）。在YouTube-8M数据集上，单个快速前向LSTM模型的GAP@20指标达到82.75%，57个模型集成后最终在650支参赛队伍中取得第三名。这项工作表明，在超大规模视频数据上深度循环网络的潜力尚未被充分挖掘，合理的网络连接设计可以有效突破深层时序模型的优化瓶颈。
原文 arXiv:1707.04555;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.04555v1