Wonder3D: Single Image to 3D using Cross-Domain Diffusion
Xiaoxiao Long1,3,6∗, Yuan-Chen Guo2,3∗, Cheng Lin1†, Yuan Liu1, Zhiyang Dou1 Lingjie Liu4, Yuexin Ma5, Song-Hai Zhang2, Marc Habermann6, Christian Theobalt6, Wenping Wang7† 1 The University of Hong Kong 2 Tsinghua University 3 VAST 4 University of Pennsylvania 5 Shanghai Tech University 6 MPI Informatik 7 Texas A、M University ∗ Equal Contributions. https://www.xxlong.site/Wonder3D/
Abstract
In this work, we introduce Wonder3D, a novel method for efficiently generating high-fidelity textured meshes from single-view images. Recent methods based on Score Distillation Sampling (SDS) have shown the potential to recover 3D geometry from 2D diffusion priors, but they typically suffer from time-consuming per-shape optimization and inconsistent geometry. In contrast, certain works directly produce 3D information via fast network inferences, but their results are often of low quality and lack geometric details. To holistically improve the quality, consistency, and efficiency of single-view reconstruction tasks, we propose a cross-domain diffusion model that generates multi-view normal maps and the corresponding color images. To ensure the consistency of generation, we employ a multi-view cross-domain attention mechanism that facilitates information exchange across views and modalities. Lastly, we introduce a geometry-aware normal fusion algorithm that extracts high-quality surfaces from the multi-view 2D representations. Our extensive evaluations demonstrate that our method achieves high-quality reconstruction results, robust generalization, and good efficiency compared to prio
中文速览
从单张图片重建出高质量的带纹理三维模型，一直是计算机视觉中的难题——现有方法要么需要耗费数十分钟乃至数小时的逐形状优化，要么虽然速度快却细节粗糙、几何失真。Wonder3D提出用一个跨域扩散模型（cross-domain diffusion model）同时生成多视角法线图（normal maps）和对应彩色图像，通过多视角跨域注意力机制（multi-view cross-domain attention）让不同视角、不同模态之间互相交换信息，从而保证多视角的几何与外观一致性；最后再用一种几何感知的法线融合算法（geometry-aware normal fusion）从这些二维表示中稳健地提取出高质量三维网格。在Google扫描物体数据集和多种风格图像上的实验表明，Wonder3D只需约两分钟便可完成重建，在几何细节、泛化能力和效率上均优于现有方法。这项工作的重要意义在于：它巧妙地将法线图与彩色图像的联合生成纳入成熟的二维扩散模型框架，无需从头训练三维模型，就能以极低的时间成本获得媲美甚至超越耗时优化方法的重建质量，为单视角三维内容创作提供了切实可行的高效方案。
原文 arXiv:2310.15008;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.15008v3