Read, Watch, and Move: Reinforcement Learning for Temporally Grounding Natural Language Descriptions in Videos
Dongliang He†, Xiang Zhao†, Jizhou Huang‡,†, Fu Li†, Xiao Liu†, Shilei Wen† †Baidu Inc., Beijing, China ‡Research Center for Social Computing and Information Retrieval, Harbin Institute of Technology, China {hedongliang01, zhaoxiang05, huangjizhou01, lifu, liuxiao12,
Abstract
The task of video grounding, which temporally localizes a natural language description in a video, plays an important role in understanding videos. Existing studies have adopted strategies of sliding window over the entire video or exhaustively ranking all possible clip-sentence pairs in a pre-segmented video, which inevitably suffer from exhaustively enumerated candidates. To alleviate this problem, we formulate this task as a problem of sequential decision making by learning an agent which regulates the temporal grounding boundaries progressively based on its policy. Specifically, we propose a reinforcement learning based framework improved by multi-task learning and it shows steady performance gains by considering additional supervised boundary information during training. Our proposed framework achieves state-of-the-art performance on ActivityNet’18 DenseCaption dataset (?) and Charades-STA dataset (?; ?) while observing only 10 or less clips per video.
中文速览
视频文本定位（video grounding）任务需要在一段未经剪辑的长视频中，精确找出与某句自然语言描述相匹配的时间片段，而现有方法往往要对整段视频做滑动窗口或穷举所有候选片段进行排序，计算代价极高。为此，研究者将该任务重新建模为序贯决策问题，训练一个基于强化学习（reinforcement learning）的智能体，让它像人一样每次只看几个片段、逐步调整时间边界，直到锁定最佳匹配区间；同时引入多任务学习，将带标注的边界回归信号作为额外监督，帮助智能体更准确地感知环境、更高效地探索。实验表明，该方法在 ActivityNet DenseCaption 和 Charades-STA 两个公认基准上达到了当时最优性能，且每个视频平均只需观看不超过10个片段。这项工作的意义在于，它从根本上打破了视频定位必须穷举候选的瓶颈，为高效、开放域的视频语言理解提供了新思路。
原文 arXiv:1901.06829;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1901.06829v1