DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation
Jiaxiang Tang1, Jiawei Ren2, Hang Zhou3, Ziwei Liu2, Gang Zeng1 1National Key Laboratory of General AI, School of IST, Peking University 2S-Lab, Nanyang Technological University 3Baidu Inc. This work was partly done when interning with Baidu Inc. and visiting NTU S-Lab.
Abstract
Recent advances in 3D content creation mostly leverage optimization-based 3D generation via score distillation sampling (SDS). Though promising results have been exhibited, these methods often suffer from slow per-sample optimization, limiting their practical usage. In this paper, we propose DreamGaussian, a novel 3D content generation framework that achieves both efficiency and quality simultaneously. Our key insight is to design a generative 3D Gaussian Splatting model with companioned mesh extraction and texture refinement in UV space. In contrast to the occupancy pruning used in Neural Radiance Fields, we demonstrate that the progressive densification of 3D Gaussians converges significantly faster for 3D generative tasks. To further enhance the texture quality and facilitate downstream applications, we introduce an efficient algorithm to convert 3D Gaussians into textured meshes and apply a fine-tuning stage to refine the details. Extensive experiments demonstrate the superior efficiency and competitive generation quality of our proposed approach. Notably, DreamGaussian produces high-quality textured meshes in just 2 minutes from a single-view image, achieving approximately 10
中文速览
从单张图片或一段文字自动生成可用的三维模型，此前主流方法要借助神经辐射场（NeRF）与分数蒸馏采样（SDS）反复优化，往往需要数小时才能出结果。DreamGaussian 改用三维高斯散射（3D Gaussian Splatting）作为优化载体，利用其"渐进式致密化"特性让生成过程比 NeRF 收敛快得多，并额外设计了一套从高斯点云提取多边形网格、再在 UV 纹理空间做细化的两阶段流程，解决了直接输出结果模糊、缺乏细节的问题。最终，该方法只需约两分钟就能从单张图像生成带有精细纹理的三维网格，速度约是现有方法的十倍，同时保持了有竞争力的生成质量。这项工作的意义在于，它把基于优化的三维内容生成从"实验室级别的耗时流程"推向了实际可部署的效率区间，为游戏、影视、元宇宙等场景的自动化三维资产创作铺平了道路。
原文 arXiv:2309.16653;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2309.16653v2