Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors
Guocheng Qian1,2, Jinjie Mai1, Abdullah Hamdi3, Jian Ren2, Aliaksandr Siarohin2, Bing Li1, Hsin-Ying Lee2, Ivan Skorokhodov1, Peter Wonka1, Sergey Tulyakov2, Bernard Ghanem1 1King Abdullah University of Science and Technology (KAUST), 2Snap Inc. 3Visual Geometry Group, University of Oxford {guocheng.qian,
Abstract
We present “Magic123”, a two-stage coarse-to-fine approach for high-quality, textured 3D meshes generation from a single unposed image in the wild using both 2D and 3D priors. In the first stage, we optimize a neural radiance field to produce a coarse geometry. In the second stage, we adopt a memory-efficient differentiable mesh representation to yield a high-resolution mesh with a visually appealing texture. In both stages, the 3D content is learned through reference view supervision and novel views guided by a combination of 2D and 3D diffusion priors. We introduce a single trade-off parameter between the 2D and 3D priors to control exploration (more imaginative) and exploitation (more precise) of the generated geometry. Additionally, we employ textual inversion and monocular depth regularization to encourage consistent appearances across views and to prevent degenerate solutions, respectively. Magic123 demonstrates a significant improvement over previous image-to-3D techniques, as validated through extensive experiments on synthetic benchmarks and diverse real-world images. Our code, models, and generated 3D assets are available at https://github.com/guochengqian/Magic123.
中文速览
从一张随手拍的普通图片自动生成高质量三维模型，是计算机视觉领域长期悬而未决的难题。Magic123 提出了一套"粗到精"两阶段流程：第一阶段用神经辐射场（NeRF）学出粗略几何，第二阶段换用内存高效的可微分网格表示（DMTet）提升分辨率并分离几何与纹理，整个过程同时引入 2D 扩散先验（Stable Diffusion）和 3D 扩散先验（Zero-1-to-3），并通过一个权衡参数在"天马行空的想象力"与"忠实的三维约束"之间灵活调节。此外，方法还借助文本反演（textual inversion）保持多视角外观一致，并用单目深度正则化防止几何坍缩成平面。在合成基准和真实野外图片上的大量实验表明，Magic123 的三维重建质量显著优于此前方法，为单图生成可用三维资产提供了一条切实可行的路径。
原文 arXiv:2306.17843;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2306.17843v2