Object Hallucination in Image Captioning
Anna Rohrbach∗1, Lisa Anne Hendricks∗1, Kaylee Burns1 , Trevor Darrell1, Kate Saenko2 1 UC Berkeley, 2 Boston University
Abstract
Despite continuously improving performance, contemporary image captioning models are prone to “hallucinating” objects that are not actually in a scene. One problem is that standard metrics only measure similarity to ground truth captions and may not fully capture image relevance. In this work, we propose a new image relevance metric to evaluate current models with veridical visual labels and assess their rate of object hallucination. We analyze how captioning model architectures and learning objectives contribute to object hallucination, explore when hallucination is likely due to image misclassification or language priors, and assess how well current sentence metrics capture object hallucination. We investigate these questions on the standard image captioning benchmark, MSCOCO, using a diverse set of models. Our analysis yields several interesting findings, including that models which score best on standard sentence metrics do not always have lower hallucination and that models which hallucinate more tend to make errors driven by language priors.
中文速览
图像自动描述（image captioning）模型虽然在各类标准指标上越来越好，却存在一个严重问题：它们会凭空"幻想"出图片中根本不存在的物体。为解决这一问题，作者提出了一种新的评估指标 CHAIR（Caption Hallucination Assessment with Image Relevance），通过对比生成描述与图像真实标注，直接衡量模型产生幻觉物体的比例。研究者对多种主流模型进行分析后发现，在传统指标上得分更高的模型并不一定幻觉更少，而且幻觉越严重的模型，其错误往往更符合语言先验（language prior）而非图像内容，说明这些模型更多在"背词共现规律"而不是真正理解图像。这项工作揭示了当前评估体系的盲区，为构建更可靠、更忠实于图像内容的描述模型提供了重要的诊断工具。
原文 arXiv:1809.02156;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1809.02156v2