To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regression
Yitian Yuan1, Tao Mei2, Wenwu Zhu1,3 1 Tsinghua-Berkeley Shenzhen Institute, Tsinghua University, China 2 JD AI Research, China 3 Department of Computer Science and Technology, Tsinghua University, China
Abstract
We have witnessed the tremendous growth of videos over the Internet, where most of these videos are typically paired with abundant sentence descriptions, such as video titles, captions and comments. Therefore, it has been increasingly crucial to associate specific video segments with the corresponding informative text descriptions, for a deeper understanding of video content. This motivates us to explore an overlooked problem in the research community — temporal sentence localization in video, which aims to automatically determine the start and end points of a given sentence within a paired video. For solving this problem, we face three critical challenges: (1) preserving the intrinsic temporal structure and global context of video to locate accurate positions over the entire video sequence; (2) fully exploring the sentence semantics to give clear guidance for localization; (3) ensuring the efficiency of the localization method to adapt to long videos. To address these issues, we propose a novel Attention Based Location Regression (ABLR) approach to localize sentence descriptions in videos in an efficient end-to-end manner. Specifically, to preserve the context information, ABLR fi
中文速览
视频在互联网上爆炸式增长，如何自动找到一段自然语言描述所对应的视频片段起止时间点（视频时序句子定位，temporal sentence localization），是一个亟待解决却长期被忽视的问题。现有方法普遍采用"滑窗扫描＋逐片段匹配"的思路，既破坏了视频的整体时序结构，又无法充分挖掘句子中关键词的细节信息，还因密集采样而效率低下。为此，作者提出了基于注意力的位置回归模型（Attention Based Location Regression，ABLR）：先用双向LSTM分别对整段视频和句子进行编码以保留上下文，再通过多模态协同注意力机制同时生成"全局视频注意力"和"关键词句子注意力"，最后由一个注意力驱动的坐标预测网络直接回归出目标片段的起止时间，全程端到端训练、无需滑窗扫描。在ActivityNet Captions和TACoS两个公开数据集上的实验表明，ABLR在定位精度和运行效率上均显著优于已有方法，为视频内容的精细语义理解提供了一条更自然、更高效的技术路径。
原文 arXiv:1804.07014;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1804.07014v4