DreamScape: 3D Scene Creation via Gaussian Splatting joint Correlation Modeling
Yueming Zhao1*, Xuening Yuan1*, Hongyu Yang2,3†, Di Huang1 1School of Computer Science and Engineering, Beihang University, Beijing, China 2School of Artificial Intelligence, Beihang University, Beijing, China 3Shanghai Artificial Intelligence Laboratory, Shanghai, China {zhaoyueming, snowing, hongyuyang,
Abstract
Recent advances in text-to-3D creation integrate the potent prior of Diffusion Models from text-to-image generation into 3D domain. Nevertheless, generating 3D scenes with multiple objects remains challenging. Therefore, we present DreamScape, a method for generating 3D scenes from text. Utilizing Gaussian Splatting for 3D representation, DreamScape introduces 3D Gaussian Guide that encodes semantic primitives, spatial transformations and relationships from text using LLMs, enabling local-to-global optimization. Progressive scale control is tailored during local object generation, addressing training instability issue arising from simple blending in the global optimization stage. Collision relationships between objects are modeled at the global level to mitigate biases in LLMs priors, ensuring physical correctness. Additionally, to generate pervasive objects like rain and snow distributed extensively across the scene, we design specialized sparse initialization and densification strategy. Experiments demonstrate that DreamScape achieves state-of-the-art performance, enabling high-fidelity, controllable 3D scene generation.
中文速览
从文字描述直接生成包含多个物体、背景和相互作用的完整三维场景，一直是三维内容生成领域的难点——现有方法要么语义理解不足导致生成混乱，要么缺乏真实的三维信息。DreamScape 的做法是先用大语言模型（LLM）解析输入文本，自动生成一份"三维高斯导引"（3D Gaussian Guide，3DG²），把场景中每个物体的类别、位置、尺寸、朝向等空间信息编码进来，再以此为骨架，分"局部-全局"两个阶段用高斯泼溅（Gaussian Splatting）逐步优化：局部阶段用渐进式尺度控制逐步把每个物体塑造到目标大小，避免几何扭曲；全局阶段引入碰撞损失来纠正 LLM 给出的空间偏差、防止物体相互穿插，同时让水面涟漪、反光等交互细节自然涌现；对雨雪这类弥漫型物体，则专门设计了稀疏初始化与定制化致密化策略。实验表明，DreamScape 在生成质量和语义一致性上达到当前最优水平，证明了把 LLM 的场景规划能力与高斯泼溅的高效渲染结合起来，是解决复杂多物体三维场景生成的一条可行路径。
原文 arXiv:2404.09227;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2404.09227v3