Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior
Junshu Tang1† Tengfei Wang2† Bo Zhang3333Corresponding authors. Ting Zhang3 Ran Yi1 Lizhuang Ma1333Corresponding authors. Dong Chen3 1Shanghai Jiao Tong University 2HKUST 3Microsoft Research https://make-it-3d.github.io/
Abstract
In this work, we investigate the problem of creating high-fidelity 3D content from only a single image. This is inherently challenging: it essentially involves estimating the underlying 3D geometry while simultaneously hallucinating unseen textures. To address this challenge, we leverage prior knowledge from a well-trained 2D diffusion model to act as 3D-aware supervision for 3D creation. Our approach, Make-It-3D, employs a two-stage optimization pipeline: the first stage optimizes a neural radiance field by incorporating constraints from the reference image at the frontal view and diffusion prior at novel views; the second stage transforms the coarse model into textured point clouds and further elevates the realism with diffusion prior while leveraging the high-quality textures from the reference image. Extensive experiments demonstrate that our method outperforms prior works by a large margin, resulting in faithful reconstructions and impressive visual quality. Our method presents the first attempt to achieve high-quality 3D creation from a single image for general objects and enables various applications such as text-to-3D creation and texture editing.
中文速览
单张图片生成高质量三维模型一直是个难题，因为仅凭一个视角既要推断几何形状，又要"脑补"看不见的纹理。Make-It-3D 借助已训练好的二维扩散模型（2D diffusion model）所蕴含的三维知识，设计了一套两阶段优化流程：第一阶段用评分蒸馏采样（Score Distillation Sampling, SDS）结合参考图的像素和深度约束，优化出一个粗粒度的神经辐射场（NeRF）；第二阶段将粗模型转为带纹理的点云，把参考图中可见区域的真实纹理直接投影过去，再用扩散先验补全遮挡区域的纹理，大幅提升视觉真实感。在公开数据集和自建400张图的测试集上，该方法显著优于此前所有方案，且不限于特定物体类别，还能支持文本生成三维模型、纹理编辑等扩展应用——这对于让普通用户也能轻松创作高质量三维内容具有重要意义。
原文 arXiv:2303.14184;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.14184v2