Learnable pooling with Context Gating for video classification
Antoine Miech, Ivan Laptev and Josef Sivic https://github.com/antoine77340/LOUPE A. Miech, I. Laptev and J. Sivic are with Inria, WILLOW, Departement d’Informatique de l’École Normale Supérieure, PSL Research University, ENS/INRIA/CNRS UMR 8548, Paris, France E-mail: {antoine.miech, ivan.laptev, J. Sivic is also with Czech Institute of Informatics, Robotics and Cybernetics, Czech Technical University in Prague.
Abstract
Current methods for video analysis often extract frame-level features using pre-trained convolutional neural networks (CNNs). Such features are then aggregated over time e.g., by simple temporal averaging or more sophisticated recurrent neural networks such as long short-term memory (LSTM) or gated recurrent units (GRU). In this work we revise existing video representations and study alternative methods for temporal aggregation. We first explore clustering-based aggregation layers and propose a two-stream architecture aggregating audio and visual features. We then introduce a learnable non-linear unit, named Context Gating, aiming to model interdependencies among network activations. Our experimental results show the advantage of both improvements for the task of video classification. In particular, we evaluate our method on the large-scale multi-modal Youtube-8M v2 dataset and outperform all other methods in the Youtube 8M Large-Scale Video Understanding challenge.
中文速览
视频内容的多标签分类面临一个核心难题：如何把一段长视频的逐帧特征高效地压缩成一个能送进分类器的整体表示。现有的循环神经网络（LSTM/GRU）虽然能捕捉时序信息，但训练慢、对长序列处理吃力，而且实验发现随机打乱帧顺序后性能几乎不变，说明时序建模并非关键所在。针对这一问题，研究者提出了两项改进：一是把基于聚类的可学习池化方法（NetVLAD、NetFV、BoW）整合进双流架构，分别处理视觉和音频特征再进行融合；二是设计了一个名为"上下文门控"（Context Gating，CG）的轻量非线性模块，让网络能自动学习不同激活维度之间的相互依赖关系，从而抑制与当前视频语境无关的特征和标签。在包含约800万个视频的超大规模YouTube-8M v2数据集上，该方法在超过650支参赛队伍中夺得第一名，证明了无需复杂时序建模、仅靠更好的聚合方式与门控机制就能显著提升大规模视频分类性能。
原文 arXiv:1706.06905;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1706.06905v2