EfficientDreamer: High-Fidelity and Stable 3D Creation via Orthogonal-view Diffusion Priors
Zhipeng Hu1††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT, Minda Zhao1††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT, Chaoyi Zhao1, Xinyue Liang1, Lincheng Li1*, Zeng Zhao1, Changjie Fan1, Xiaowei Zhou2, Xin Yu3 1NetEase Fuxi AI Lab, 2State Key Lab of CAD、CG, Zhejiang University 3University of Queensland
Abstract
While image diffusion models have made significant progress in text-driven 3D content creation, they often fail to accurately capture the intended meaning of text prompts, especially for view information. This limitation leads to the Janus problem, where multi-faced 3D models are generated under the guidance of such diffusion models. In this paper, we propose a robust high-quality 3D content generation pipeline by exploiting orthogonal-view image guidance. First, we introduce a novel 2D diffusion model that generates an image consisting of four orthogonal-view sub-images based on the given text prompt. Then, the 3D content is created using this diffusion model. Notably, the generated orthogonal-view image provides strong geometric structure priors and thus improves 3D consistency. As a result, it effectively resolves the Janus problem and significantly enhances the quality of 3D content creation. Additionally, we present a 3D synthesis fusion network that can further improve the details of the generated 3D contents. Both quantitative and qualitative evaluations demonstrate that our method surpasses previous text-to-3D techniques. Project page: https://efficientdreamer.github.io.
中文速览
用文本描述生成三维内容时，长期困扰研究者的"Janus问题"（即生成的三维模型出现多个正面、视角混乱）根源在于二维扩散模型无法准确理解视角指令。EfficientDreamer针对这一痛点，提出了一种正交视图扩散模型（orthogonal-view diffusion model），在给定文本提示后，该模型能同时生成一张包含前、后、左、右四个正交视角子图的拼合图像，从而为三维生成提供强几何结构先验。在此基础上，研究者设计了一个三维合成融合网络（3D synthesis fusion network），将正交视图扩散先验与原始预训练二维扩散先验动态结合——先期着重正交视图引导以快速建立一致的几何结构，后期逐步增大二维扩散先验的权重以细化纹理细节。定量和定性实验均表明，EfficientDreamer在生成质量和视角一致性上超越了此前主流的文本到三维方法，为高保真三维内容创作提供了更稳定可靠的解决方案。
原文 arXiv:2308.13223;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.13223v2