ComboVerse: Compositional 3D Assets Creation Using Spatially-Aware Diffusion Guidance
Yongwei Chen111Equal contribution.222Work done when interning at Shanghai Artificial Intelligence Laboratory. 11 Tengfei Wang111Equal contribution. 22 Tong Wu 33 Xingang Pan 11 Kui Jia 44 Ziwei Liu 11
Abstract
Generating high-quality 3D assets from a given image is highly desirable in various applications such as AR/VR. Recent advances in single-image 3D generation explore feed-forward models that learn to infer the 3D model of an object without optimization. Though promising results have been achieved in single object generation, these methods often struggle to model complex 3D assets that inherently contain multiple objects. In this work, we present ComboVerse, a 3D generation framework that produces high-quality 3D assets with complex compositions by learning to combine multiple models. 1) We first perform an in-depth analysis of this “multi-object gap” from both model and data perspectives. 2) Next, with reconstructed 3D models of different objects, we seek to adjust their sizes, rotation angles, and locations to create a 3D asset that matches the given image. 3) To automate this process, we apply spatially-aware score distillation sampling (SSDS) from pretrained diffusion models to guide the positioning of objects. Our proposed framework emphasizes spatial alignment of objects, compared with standard score distillation sampling, and thus achieves more accurate results. Extensive exp
中文速览
单张照片里有多个物体时，现有的图像转3D模型往往因训练数据（Objaverse）几乎全是单体资产而频繁失败——物体之间出现几何"渗色"、遮挡处理混乱、摄像机假设也不再成立。ComboVerse 的解法是"分而治之再拼合"：先用分割与图像修复把每个物体单独抠出、补全遮挡区域，再分别重建成独立3D模型，最后通过优化每个模型的缩放、旋转和位移参数把它们拼成完整资产；为了让拼合位置准确，论文还提出了空间感知得分蒸馏采样（Spatially-Aware SDS），通过在扩散模型注意力图中上调位置关系词的权重，使预训练扩散模型真正"关注"物体的空间摆放而非仅仅内容匹配。在自建的100张复杂场景基准上，ComboVerse 在多物体、遮挡和相机设置三类难点上均明显优于现有方法，为将对象级3D生成能力扩展到组合场景提供了一条可行路径。
原文 arXiv:2403.12409;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2403.12409v1