Multimodal Convolutional Neural Networks for Matching Image and Sentence
Lin Ma Zhengdong Lu Lifeng Shang Hang Li Noah’s Ark Lab, Huawei Technologies {Lu.Zhengdong, Shang.Lifeng,
Abstract
In this paper, we propose multimodal convolutional neural networks ( $m$ -CNNs) for matching image and sentence. Our $m$ -CNN provides an end-to-end framework with convolutional architectures to exploit image representation, word composition, and the matching relations between the two modalities. More specifically, it consists of one image CNN encoding the image content, and one matching CNN learning the joint representation of image and sentence. The matching CNN composes words to different semantic fragments and learns the inter-modal relations between image and the composed fragments at different levels, thus fully exploit the matching relations between image and sentence. Experimental results on benchmark databases of bidirectional image and sentence retrieval demonstrate that the proposed $m$ -CNNs can effectively capture the information necessary for image and sentence matching. Specifically, our proposed $m$ -CNNs for bidirectional image and sentence retrieval on Flickr30K and Microsoft COCO databases achieve the state-of-the-art performances.
中文速览
图像与自然语言句子的语义匹配长期面临一个核心难题：二者之间的对应关系同时发生在词、短语、句子等多个粒度层次，而现有方法往往只顾其中一个层次，无法全面捕捉这种多层次的跨模态关联。为此，研究者提出了多模态卷积神经网络（multimodal CNN，m-CNN），用一个端到端的卷积框架同时处理图像编码、词语组合以及跨模态匹配：图像侧用CNN提取特征，文本侧用卷积层将词逐步组合成短语乃至完整句子，再让图像表示与不同粒度的语言片段在各层次上交互，从而联合学习从词级到句级的匹配关系。在Flickr30K和Microsoft COCO两个主流基准上的双向图文检索实验表明，m-CNN在当时达到了最优性能。这项工作的意义在于首次将卷积网络系统地引入图文匹配任务，证明了端到端多层次跨模态建模的有效性，为后续视觉-语言理解研究奠定了重要基础。
原文 arXiv:1504.06063;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1504.06063v5