TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answering
Yushi Hu1 Benlin Liu1 Jungo Kasai1 Yizhong Wang1 Mari Ostendorf1 Ranjay Krishna1,2 Noah A. Smith1,2 1University of Washington 2Allen Institute for AI https://tifa-benchmark.github.io/
Abstract
Despite thousands of researchers, engineers, and artists actively working on improving text-to-image generation models, systems often fail to produce images that accurately align with the text inputs. We introduce TIFA (Text-to-Image Faithfulness evaluation with question Answering), an automatic evaluation metric that measures the faithfulness of a generated image to its text input via visual question answering (VQA). Specifically, given a text input, we automatically generate several question-answer pairs using a language model. We calculate image faithfulness by checking whether existing VQA models can answer these questions using the generated image. TIFA is a reference-free metric that allows for fine-grained and interpretable evaluations of generated images. TIFA also has better correlations with human judgments than existing metrics. Based on this approach, we introduce TIFA v1.0, a benchmark consisting of 4K diverse text inputs and 25K questions across 12 categories (object, counting, etc.). We present a comprehensive evaluation of existing text-to-image models using TIFA v1.0 and highlight the limitations and challenges of current models. For instance, we find that current
中文速览
准确评估文生图（text-to-image）模型生成的图像是否真正符合文字描述，一直缺乏可靠的自动化方法——现有的CLIP相似度或图像描述比对等指标既不精细也和人类判断相关性低。研究团队提出了TIFA（基于视觉问答的文图忠实度评估），核心思路是：先用GPT-3从输入文本自动生成一批涵盖物体、数量、颜色、空间关系等12个类别的问答对，再让VQA模型用生成图像来回答这些问题，以答题准确率衡量图像与文本的匹配程度。基于此，他们构建了包含4000条多样化文本提示和2.5万道问题的TIFA v1.0基准，系统评测了多个主流文生图模型，发现这些模型虽然在常见物体和颜色渲染上表现尚可，但在数量计数、空间关系和多物体组合方面仍有明显短板；与人类判断的相关性（Spearman ρ=0.60）也显著优于CLIPScore（ρ=0.33）等已有指标。这项工作为整个领域提供了一把更精准、可解释、且随VQA技术进步可持续升级的评测尺子，有助于清晰定位当前模型的弱点并指导后续研究方向。
原文 arXiv:2303.11897;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.11897v3