Simple image description generator via a linear phrase-based model
Rémi Lebret∗、Pedro O. Pinheiro Idiap Research Institute, Martigny, Switzerland École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland、Ronan Collobert Facebook AI Research, Menlo Park, CA, USA Idiap Research Institute, Martigny, Switzerland These two authors contributed equally to this work.All research was conducted at the Idiap Research Institute, before Ronan Collobert joined Facebook AI Research.
Abstract
Generating a novel textual description of an image is an interesting problem that connects computer vision and natural language processing. In this paper, we present a simple model that is able to generate descriptive sentences given a sample image. This model has a strong focus on the syntax of the descriptions. We train a purely bilinear model that learns a metric between an image representation (generated from a previously trained Convolutional Neural Network) and phrases that are used to described them. The system is then able to infer phrases from a given image sample. Based on caption syntax statistics, we propose a simple language model that can produce relevant descriptions for a given test image using the phrases inferred. Our approach, which is considerably simpler than state-of-the-art models, achieves comparable results on the recently release Microsoft COCO dataset.
中文速览
图像自动描述生成一直是连接计算机视觉与自然语言处理的难题，现有方法普遍依赖结构复杂的循环神经网络（RNN/LSTM）。本文提出了一种基于短语（phrase-based）的轻量级方案：先用预训练卷积神经网络（CNN）提取图像特征，再训练一个双线性度量模型，将图像表示与名词短语、动词短语、介词短语的词向量均值对齐到同一空间，从而为给定图像预测最可能出现的短语；随后借助对标注句子句法结构的统计规律，用一个三元语言模型将预测短语拼接成完整描述句，并以图像与句子向量的点积得分选出最终输出。在微软COCO数据集上，该方法的BLEU评分与当时最优的深度循环网络方案相当，甚至接近人类标注者之间的一致性得分。这一结果说明，不必堆叠复杂的序列模型，抓住图像描述句子天然的短语组合结构就能以更简洁的架构达到有竞争力的效果，为后续更轻量的图文生成研究提供了有益参考。
原文 arXiv:1412.8419;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1412.8419v3