Weakly-Supervised Alignment of Video With Text
P. Bojanowski1, R. Lajugie1, E. Grave2, F. Bach1, I. Laptev1, J. Ponce3, C. Schmid1, 1INRIA 2Columbia University 3ENS / PSL Research University WILLOW project-team, Département d’Informatique de l’Ecole Normale Supérieure, ENS/INRIA/CNRS UMR 8548, Paris, France.SIERRA project-team, Département d’Informatique de l’Ecole Normale Supérieure, ENS/INRIA/CNRS UMR 8548, Paris, France.Department of Applied Physics、Applied Mathematics, Columbia University, New York, NY, USA.LEAR project-team, INRIA Grenoble Rhône-Alpes, Laboratoire Jean Kuntzmann, CNRS, Univ. Grenoble Alpes, France
Abstract
Suppose that we are given a set of videos, along with natural language descriptions in the form of multiple sentences (e.g., manual annotations, movie scripts, sport summaries etc.), and that these sentences appear in the same temporal order as their visual counterparts. We propose in this paper a method for aligning the two modalities, i.e., automatically providing a time (frame) stamp for every sentence. Given vectorial features for both video and text, this can be cast as a temporal assignment problem, with an implicit linear mapping between the two feature modalities. We formulate this problem as an integer quadratic program, and solve its continuous convex relaxation using an efficient conditional gradient algorithm. Several rounding procedures are proposed to construct the final integer solution. After demonstrating significant improvements over the state of the art on the related task of aligning video with symbolic labels [7], we evaluate our method on a challenging dataset of videos with associated textual descriptions [37], and explore bag-of-words and continuous representations for text.
中文速览
视频和文字在时间轴上自动对齐——比如把一段烹饪视频的每一帧和对应的步骤说明句子匹配起来——是弱监督视频理解的核心难题。研究者将这一对齐任务建模为整数二次规划问题：在保持时序顺序的前提下，同时学习一个视频特征到文本特征的线性映射，并求解最优的帧-句子分配矩阵。为了让求解可行，他们对离散问题做凸松弛，用 Frank-Wolfe 条件梯度算法高效求解，再通过多种取整策略还原整数解；同时引入时长先验和路径带约束来避免退化解。在符号标签对齐基准上大幅超越已有方法，在真实烹饪视频配文字描述数据集上也取得了有竞争力的结果，验证了该框架既能处理离散词袋表示，也能处理连续词向量，为用自然语言句子作为弱监督信号来理解长视频提供了一条切实可行的路径。
原文 arXiv:1505.06027;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1505.06027v2