Frankenstein: Generating Semantic-Compositional 3D Scenes in One Tri-Plane
Han Yan Shanghai Jiao Tong UniversityShanghaiChina , Yang Li Tencent XR Vision LabsShanghaiChina , Zhennan Wu The University of TokyoTokyoJapan , Shenzhou Chen Tencent XR Vision LabsShanghaiChina , Weixuan Sun Tencent XR Vision LabsShanghaiChina , Taizhang Shang Tencent XR Vision LabsShanghaiChina , Weizhe Liu Tencent XR Vision LabsShanghaiChina , Tian Chen Tencent XR Vision LabsShanghaiChina , Xiaqiang Dai Tencent XR Vision LabsShanghaiChina , Chao Ma Shanghai Jiao Tong UniversityShanghaiChina , Hongdong Li Australian National UniversityCanberraAustralia and Pan Ji Tencent XR Vision LabsShanghaiChina
Abstract
Abstract: We present Frankenstein, a diffusion-based framework that can generate semantic-compositional 3D scenes in a single pass. Unlike existing methods that output a single, unified 3D shape, Frankenstein simultaneously generates multiple separated shapes, each corresponding to a semantically meaningful part. The 3D scene information is encoded in one single tri-plane tensor, from which multiple Signed Distance Function (SDF) fields can be decoded to represent the compositional shapes. During training, an auto-encoder compresses tri-planes into a latent space, and then the denoising diffusion process is employed to approximate the distribution of the compositional scenes. Frankenstein demonstrates promising results in generating room interiors as well as human avatars with automatically separated parts. The generated scenes facilitate many downstream applications, such as part-wise re-texturing, object rearrangement in the room or avatar cloth re-targeting. Our project page is available at:https://wolfball.github.io/frankenstein/.
中文速览
用一张"三平面"（tri-plane）张量同时编码多个语义部件的完整三维形状，是这项工作想攻克的核心难题——传统方法只能生成一个混在一起的整体三维场，后续再分割往往产生破碎、不完整的部件。Frankenstein 的做法是：在单张三平面里解码出多个独立的符号距离场（SDF），每个 SDF 对应一类语义部件（如房间里的床、柜子、墙，或虚拟人的身体、头发、服装），再通过变分自编码器把三平面压缩到紧凑的隐空间，最后训练一个去噪扩散模型（denoising diffusion model）在该隐空间中一次前向推理就生成完整的语义组合场景。实验表明，模型能生成布局合理、部件形状完整的室内场景与可组合的数字人，并直接支持按部件重贴图、家具重摆放、服装迁移等下游应用。这项工作的价值在于，它首次将语义可分解的三维内容生成统一进单次扩散推理流程，无需繁琐的逐对象优化或额外的分割后处理，为三维内容创作的自动化和可编辑性打开了新的可能。
原文 arXiv:2403.16210;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2403.16210v2