Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)
Junhua Mao University of California, Los Angeles; Baidu Research、Wei Xu、Yi Yang、Jiang Wang、Zhiheng Huang Baidu Research \ANDAlan Yuille University of California, Los Angeles
Abstract
In this paper, we present a multimodal Recurrent Neural Network (m-RNN) model for generating novel image captions. It directly models the probability distribution of generating a word given previous words and an image. Image captions are generated according to this distribution. The model consists of two sub-networks: a deep recurrent neural network for sentences and a deep convolutional network for images. These two sub-networks interact with each other in a multimodal layer to form the whole m-RNN model. The effectiveness of our model is validated on four benchmark datasets (IAPR TC-12, Flickr 8K, Flickr 30K and MS COCO). Our model outperforms the state-of-the-art methods. In addition, we apply the m-RNN model to retrieval tasks for retrieving images or sentences, and achieves significant performance improvement over the state-of-the-art methods which directly optimize the ranking objective function for retrieval. The project page of this work is: www.stat.ucla.edu/~junhua.mao/m-RNN.html. 111Most recently, we adopt a simple strategy to boost the performance of image captioning task significantly. More details are shown in Section 8. The code and related data (e.g. refined image f
中文速览
图像自动描述生成一直面临两大难题：既要能写出从未见过的新句子，又要能做好图文互检索。这项工作提出了一种多模态循环神经网络（multimodal RNN，m-RNN），把专门处理语言的循环网络和专门提取图像特征的卷积网络（CNN）通过一个多模态融合层"接"在一起，让模型在生成每个词时都能同时参考前文语境和图像内容，从而直接对"给定图像和上文、下一个词是什么"这一概率分布建模。与此前方法相比，m-RNN采用了双层词嵌入和ReLU激活函数，解决了梯度消失问题，并在每个时间步都向模型输入图像信息而非仅在开头输入一次，更充分地利用了循环层的容量。在IAPR TC-12、Flickr 8K、Flickr 30K、MS COCO四个主流基准上，m-RNN在图像描述生成、以句检图、以图检句三项任务中均超越了当时所有已有方法，包括那些专门针对检索目标函数优化的模型。这一工作的意义在于证明了将语言模型与视觉模型深度融合、端到端训练的路线可以同时兼顾生成与检索两类任务，为后续图文多模态研究奠定了重要基础。
原文 arXiv:1412.6632;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1412.6632v5