Diffusion-Based Scene Graph to Image Generation with Masked Contrastive Pre-Training
Ling Yang1∗ Zhilin Huang2 Yang Song3 Shenda Hong1 Guohao Li4 Wentao Zhang5 Bin Cui1 Bernard Ghanem4 Ming-Hsuan Yang6,7 1Peking University 2Tsinghua University 3OpenAI 4KAUST 5Mila 6University of California, Merced 7Google Research {hongshenda, {guohao.li, Contributed equally.
Abstract
Generating images from graph-structured inputs, such as scene graphs, is uniquely challenging due to the difficulty of aligning nodes and connections in graphs with objects and their relations in images. Most existing methods address this challenge by using scene layouts, which are image-like representations of scene graphs designed to capture the coarse structures of scene images. Because scene layouts are manually crafted, the alignment with images may not be fully optimized, causing suboptimal compliance between the generated images and the original scene graphs. To tackle this issue, we propose to learn scene graph embeddings by directly optimizing their alignment with images. Specifically, we pre-train an encoder to extract both global and local information from scene graphs that are predictive of the corresponding images, relying on two loss functions: masked autoencoding loss and contrastive loss. The former trains embeddings by reconstructing randomly masked image regions, while the latter trains embeddings to discriminate between compliant and non-compliant images according to the scene graph. Given these embeddings, we build a latent diffusion model to generate images fro
中文速览
如何让AI真正"读懂"场景图（scene graph）并据此生成图像，是一个棘手难题：现有方法依赖人工设计的场景布局（scene layout）作为中间桥梁，但这类布局无法精确表达"eating""looking at"等抽象关系，与图像的对齐也并不充分。SGDiff提出直接通过自监督学习来训练场景图编码器：一方面用遮蔽自编码损失（masked autoencoding loss）让嵌入向量学会预测图像中被遮挡的局部区域，另一方面用对比学习损失（contrastive loss）让嵌入向量区分哪些图像真正符合场景图描述，从而同时捕获局部细节和全局结构信息。有了这套嵌入表示，再结合潜在扩散模型（latent diffusion model）来完成场景图到图像的生成，并支持通过修改场景图节点和边来语义化地操控生成结果。在Visual Genome和COCO-Stuff两个标准数据集上，SGDiff在IS和FID指标上均超越了现有最优方法，证明了绕开人工布局、直接优化图-图像对齐是一条更有效的技术路线。
原文 arXiv:2211.11138;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2211.11138v1