Semantic Score Distillation Sampling for Compositional Text-to-3D Generation
Ling Yang1†, Zixiang Zhang, Junlin Han 2, Bohan Zeng1, Runjia Li2 Philip Torr2, Wentao Zhang1 1Peking University 2University of Oxford Project: https://github.com/YangLing0818/SemanticSDS-3D Contributed equally.Corresponding authors:
Abstract
Generating high-quality 3D assets from textual descriptions remains a pivotal challenge in computer graphics and vision research. Due to the scarcity of 3D data, state-of-the-art approaches utilize pre-trained 2D diffusion priors, optimized through Score Distillation Sampling (SDS). Despite progress, crafting complex 3D scenes featuring multiple objects or intricate interactions is still difficult. To tackle this, recent methods have incorporated box or layout guidance. However, these layout-guided compositional methods often struggle to provide fine-grained control, as they are generally coarse and lack expressiveness. To overcome these challenges, we introduce a novel SDS approach, Semantic Score Distillation Sampling (SemanticSDS), designed to effectively improve the expressiveness and accuracy of compositional text-to-3D generation. Our approach integrates new semantic embeddings that maintain consistency across different rendering views and clearly differentiate between various objects and parts. These embeddings are transformed into a semantic map, which directs a region-specific SDS process, enabling precise optimization and compositional generation. By leveraging explicit s
中文速览
文本生成3D场景一直面临多物体、复杂交互难以精细控制的痛点——现有方法靠粗粒度的包围盒布局来引导优化，根本管不住细节。为此，研究者提出了语义分数蒸馏采样（Semantic Score Distillation Sampling，SemanticSDS），核心思路是给每个3D高斯点云附上由CLIP文本编码器生成的语义嵌入，渲染时将这些嵌入投影成语义图，再以区域级别的方式驱动分数蒸馏采样（Score Distillation Sampling，SDS）优化，让不同物体和部件各自受到对应文字描述的精准监督。此外，方法还引入程序辅助的布局规划，让大语言模型通过Python解释器把模糊的自然语言描述转化为精确的3D坐标，有效避免物体重叠或错位。实验结果表明，SemanticSDS在生成包含多个物体、复杂属性和精细交互的3D内容时显著优于现有方法，为文本驱动的复杂3D场景生成树立了新的质量标杆。
原文 arXiv:2410.09009;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2410.09009v1