CIDEr: Consensus-based Image Description Evaluation
Ramakrishna Vedantam Virginia Tech C. Lawrence Zitnick Microsoft Research Devi Parikh Virgnia Tech
Abstract
Automatically describing an image with a sentence is a long-standing challenge in computer vision and natural language processing. Due to recent progress in object detection, attribute classification, action recognition, etc., there is renewed interest in this area. However, evaluating the quality of descriptions has proven to be challenging. We propose a novel paradigm for evaluating image descriptions that uses human consensus. This paradigm consists of three main parts: a new triplet-based method of collecting human annotations to measure consensus, a new automated metric (CIDEr) that captures consensus, and two new datasets: PASCAL-50S and ABSTRACT-50S that contain 50 sentences describing each image. Our simple metric captures human judgment of consensus better than existing metrics across sentences generated by various sources. We also evaluate five state-of-the-art image description approaches using this new protocol and provide a benchmark for future comparisons. A version of CIDEr named CIDEr-D is available as a part of MS COCO evaluation server to enable systematic evaluation and benchmarking.
中文速览
自动生成图像描述句子的质量很难评估——现有指标如BLEU、ROUGE与人类判断相关性较弱，且已有数据集每张图片只有5句描述，难以代表"大多数人"的共识。为此，作者提出了一套基于"共识"（consensus）的评估框架：通过让人类标注者回答"B和C两句话哪句更像A"这种三元组强迫选择任务来收集判断标准，并在此基础上设计了自动评估指标CIDEr（Consensus-based Image Description Evaluation），该指标用TF-IDF加权的n-gram余弦相似度来衡量候选句子与多条人工参考句的吻合程度。同时，作者发布了两个每张图片含50条描述的新数据集PASCAL-50S和ABSTRACT-50S，实验表明参考句子越多，各类指标的准确性越高，而CIDEr在所有测试条件下与人类共识判断的一致性均优于现有同类指标。这项工作为图像描述生成领域提供了更可靠的自动评估标准，CIDEr-D已被集成到MS COCO评测服务器，对推动该领域的系统性比较与进步具有重要意义。
原文 arXiv:1411.5726;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1411.5726v2