A Hierarchical Approach for Generating Descriptive Image Paragraphs
Jonathan Krause Justin Johnson Ranjay Krishna Li Fei-Fei Stanford University
Abstract
Recent progress on image captioning has made it possible to generate novel sentences describing images in natural language, but compressing an image into a single sentence can describe visual content in only coarse detail. While one new captioning approach, dense captioning, can potentially describe images in finer levels of detail by captioning many regions within an image, it in turn is unable to produce a coherent story for an image. In this paper we overcome these limitations by generating entire paragraphs for describing images, which can tell detailed, unified stories. We develop a model that decomposes both images and paragraphs into their constituent parts, detecting semantic regions in images and using a hierarchical recurrent neural network to reason about language. Linguistic analysis confirms the complexity of the paragraph generation task, and thorough experiments on a new dataset of image and paragraph pairs demonstrate the effectiveness of our approach.
中文速览
用一段文字描述一张图片往往太简略，而逐区域标注虽然细节丰富却缺乏整体连贯性——这篇论文提出了"图像段落生成"（image paragraph generation）任务，让模型为一张图片自动写出一段完整、连贯且细节丰富的自然语言描述。作者设计了一套分层架构：先用区域检测器从图像中提取多个语义区域的特征并做最大池化聚合，再送入由"句子级RNN"和"词级RNN"构成的层级循环神经网络，由句子RNN决定段落由几句话组成并为每句生成主题向量，再由词级RNN逐词生成每句内容。为支撑研究，他们还新建了一个包含近两万张图片及配对段落描述的数据集，并通过语言学分析证明段落在多样性、动词/代词比例等方面显著优于单句描述。实验表明，这一分层模型在自动评价指标和人工评估上均明显优于直接生成长句的基线方法，为需要细节又追求连贯性的图像描述任务提供了切实可行的解决方案。
原文 arXiv:1611.06607;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1611.06607v2