Learning to Discretely Compose Reasoning Module Networks for Video Captioning
Ganchao Tan1∗ Daqing Liu1111Equal Contribution Meng Wang2、Zheng-Jun Zha1222Corresponding Author 1University of Science and Technology of China 2Hefei University of Technology {tgc1997,
Abstract
Generating natural language descriptions for videos, i.e., video captioning, essentially requires step-by-step reasoning along the generation process. For example, to generate the sentence “a man is shooting a basketball”, we need to first locate and describe the subject “man”, next reason out the man is “shooting”, then describe the object “basketball” of shooting. However, existing visual reasoning methods designed for visual question answering are not appropriate to video captioning, for it requires more complex visual reasoning on videos over both space and time, and dynamic module composition along the generation process. In this paper, we propose a novel visual reasoning approach for video captioning, named Reasoning Module Networks (RMN), to equip the existing encoder-decoder framework with the above reasoning capacity. Specifically, our RMN employs 1) three sophisticated spatio-temporal reasoning modules, and 2) a dynamic and discrete module selector trained by a linguistic loss with a Gumbel approximation. Extensive experiments on MSVD and MSR-VTT datasets demonstrate the proposed RMN outperforms the state-of-the-art methods while providing an explicit and explainable gene
中文速览
视频自动描述生成（video captioning）不只是"看图说话"，而是需要按照"先找主体、再推动作、再定宾语"这样一步步推理才能造出准确的句子，但现有方法大多忽略了这一推理本质。作者提出了"推理模块网络"（Reasoning Module Networks，RMN），在经典的编码器-解码器框架里嵌入三个专用的时空推理模块：Locate模块负责在视频的时间和空间维度上定位单个区域以生成名词，Relate模块通过跨帧的两两配对交互来捕捉动作信息以生成动词，Func模块则只依赖语言上下文生成虚词；同时设计了一个离散动态模块选择器，借助Gumbel-Softmax技巧和词性标注监督信号，让整个模型可以端到端训练，并在生成每个词时自动决定调用哪个模块。在MSVD和MSR-VTT两个标准数据集上的实验表明，RMN在多项评测指标上超越了当时最优方法，并且每一步用了哪个推理模块都清晰可见，使生成过程具备良好的可解释性，为视频理解与语言生成的深度结合提供了一条新路径。
原文 arXiv:2007.09049;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.09049v1