REPARO: Compositional 3D Assets Generation with Differentiable 3D Layout Alignment
Haonan Han Rui Yang Huan Liao Jiankai Xing1 Zunnan Xu1 Xiaoming Yu3 Junwei Zha3 Xiu Li Wanhua Li 1 Tsinghua University 2 The University of Hong Kong 3 Tencent 4 Harvard University
Abstract
Traditional image-to-3D models often struggle with scenes containing multiple objects due to biases and occlusion complexities. To address this challenge, we present REPARO, a novel approach for compositional 3D asset generation from single images. REPARO employs a two-step process: first, it extracts individual objects from the scene and reconstructs their 3D meshes using image-to-3D models; then, it optimizes the layout of these meshes through differentiable rendering techniques, ensuring coherent scene composition. By integrating optimal transport-based long-range appearance loss term and high-level semantic loss term in the differentiable rendering, REPARO can effectively recover the layout of 3D assets. The proposed method can significantly enhance object independence, detail accuracy, and overall scene coherence. Extensive evaluation of multi-object scenes demonstrates that our REPARO offers a comprehensive approach to address the complexities of multi-object 3D scene generation from single images. The demo have been available at https://reparo-3d.github.io/.
中文速览
多目标场景的单图三维重建一直是个老大难问题——现有image-to-3D模型因训练数据偏差和遮挡处理不当，往往把多个物体"糊"成一个错误的整体。REPARO通过"先分后合"的两阶段流程来解决这个问题：第一阶段用分割和图像修复把场景里的每个物体单独抠出来，再交给现成的image-to-3D模型分别生成高质量三维网格；第二阶段将所有网格放入同一坐标系，用可微渲染（differentiable rendering）优化每个物体的位置、旋转和缩放，核心创新在于引入了基于最优传输（optimal transport）的长程外观损失——同时考虑RGB颜色、深度和像素位置来建立全局对应关系，避免像素级L2损失陷入局部最优——以及高层语义损失来保证场景整体语义一致。在Google Scanned Objects数据集的多物体场景上，REPARO在物体独立性、几何细节和场景布局一致性等方面均显著优于现有方法，为AR/VR、游戏等领域的多目标三维内容自动生成提供了一条实用可行的新路径。
原文 arXiv:2405.18525;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2405.18525v2