LucidDreamer: Towards High-Fidelity Text-to-3D Generation via Interval Score Matching
Yixun Liang*1*1{}^{\text{{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}*}}}1} Xin Yang*1,2*12{}^{\text{{{\color[rgb]{1,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{1,0,0}*}}}1,2} Jiantao Lin1 Haodong Li1 Xiaogang Xu3,4 Yingcong Chen∗∗1,2 1 HKUST (GZ) 2 HKUST 3 Zhejiang Lab 4 Zhejiang University
Abstract
The recent advancements in text-to-3D generation mark a significant milestone in generative models, unlocking new possibilities for creating imaginative 3D assets across various real-world scenarios. While recent advancements in text-to-3D generation have shown promise, they often fall short in rendering detailed and high-quality 3D models. This problem is especially prevalent as many methods base themselves on Score Distillation Sampling (SDS). This paper identifies a notable deficiency in SDS, that it brings inconsistent and low-quality updating direction for the 3D model, causing the over-smoothing effect. To address this, we propose a novel approach called Interval Score Matching (ISM). ISM employs deterministic diffusing trajectories and utilizes interval-based score matching to counteract over-smoothing. Furthermore, we incorporate 3D Gaussian Splatting into our text-to-3D generation pipeline. Extensive experiments show that our model largely outperforms the state-of-the-art in quality and training efficiency. Our code will be available at: EnVision-Research/LucidDreamer
中文速览
文本生成3D模型（text-to-3D generation）领域长期受困于"过度平滑（over-smoothing）"问题，生成结果细节模糊、质量偏低，根源在于主流方法所依赖的得分蒸馏采样（Score Distillation Sampling, SDS）会产生前后不一致、质量低劣的"伪真值"，把这些杂乱的更新信号平均叠加到3D模型上，最终把细节都抹平了。为此，论文提出"区间得分匹配（Interval Score Matching, ISM）"：先用DDIM反转构造一条确定性的扩散轨迹，消除随机噪声带来的伪真值不一致问题；再把原来的单步预测改为在扩散轨迹的两个区间步之间做匹配，从根本上避免单步重建误差大的缺陷。在此基础上，论文还将3D高斯散射（3D Gaussian Splatting）引入生成流程，实现高效渲染。大量实验表明，该方法在生成质量和训练效率上均显著超越Magic3D、ProlificDreamer等当前最优方法，且无需多阶段训练，流程更简洁。
原文 arXiv:2311.11284;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.11284v3