One-2-3-45++: Fast Single Image to 3D Objects with Consistent Multi-View Generation and 3D Diffusion
Minghua Liu1 Ruoxi Shi1* Linghao Chen1,2* Zhuoyang Zhang3*††{\dagger} Chao Xu4* Xinyue Wei1 Hansheng Chen5††{\dagger} Chong Zeng2††{\dagger} Jiayuan Gu1 Hao Su1 1 UC San Diego 2 Zhejiang University 3 Tsinghua University 4 UCLA 5 Stanford University Equal contribution.Work done during internship at UC San Diego.
Abstract
Recent advancements in open-world 3D object generation have been remarkable, with image-to-3D methods offering superior fine-grained control over their text-to-3D counterparts. However, most existing models fall short in simultaneously providing rapid generation speeds and high fidelity to input images - two features essential for practical applications. In this paper, we present One-2-3-45++, an innovative method that transforms a single image into a detailed 3D textured mesh in approximately one minute. Our approach aims to fully harness the extensive knowledge embedded in 2D diffusion models and priors from valuable yet limited 3D data. This is achieved by initially finetuning a 2D diffusion model for consistent multi-view image generation, followed by elevating these images to 3D with the aid of multi-view conditioned 3D native diffusion models. Extensive experimental evaluations demonstrate that our method can produce high-quality, diverse 3D assets that closely mirror the original input image. Our project webpage: https://sudo-ai-3d.github.io/One2345plus_page.
中文速览
单张图片生成高质量三维网格模型，难点在于现有方法要么速度慢（逐形状优化动辄几十分钟），要么多视角预测不一致导致三维重建质量差。One-2-3-45++ 将生成拆成两步：先把六个视角的图像拼成一张大图，用微调后的2D扩散模型（Stable Diffusion）一次性生成全部视角，从而让各视角在扩散过程中相互"看到"彼此，保证多视角一致性；再把这批一致的多视角图像送入一个以三维数据为训练基础的3D扩散模型，分粗到细预测带颜色的SDF体素，最终提取出有纹理的网格，并经过轻量级纹理优化收尾。实验和用户研究表明，整个流程不到一分钟即可完成，生成的三维资产在几何准确性、纹理质量和对输入图像的忠实度上均优于此前方法。这项工作证明了把2D大模型的丰富先验与有限但高质量的3D数据结合起来是一条切实可行的路径，对游戏、影视等需要快速生产三维内容的行业具有直接应用价值。
原文 arXiv:2311.07885;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.07885v1