Shap⋅⋅\cdotE: Generating Conditional 3D Implicit Functions
Heewoo Jun、Alex Nichol * Equal contribution
Abstract
We present Shap $\cdot$ E, a conditional generative model for 3D assets. Unlike recent work on 3D generative models which produce a single output representation, Shap $\cdot$ E directly generates the parameters of implicit functions that can be rendered as both textured meshes and neural radiance fields. We train Shap $\cdot$ E in two stages: first, we train an encoder that deterministically maps 3D assets into the parameters of an implicit function; second, we train a conditional diffusion model on outputs of the encoder. When trained on a large dataset of paired 3D and text data, our resulting models are capable of generating complex and diverse 3D assets in a matter of seconds. When compared to Point $\cdot$ E, an explicit generative model over point clouds, Shap $\cdot$ E converges faster and reaches comparable or better sample quality despite modeling a higher-dimensional, multi-representation output space. We release model weights, inference code, and samples at https://github.com/openai/shap-e.
中文速览
三维内容生成领域长期面临一个难题：如何高效地生成既能渲染成逼真图像、又能直接用于下游三维应用的高质量3D资产。Shap·E 采用两阶段训练策略来解决这一问题：第一阶段训练一个基于Transformer的编码器，将3D资产确定性地映射为隐式函数（implicit function）的参数，使其同时支持神经辐射场（NeRF）渲染和带纹理网格（textured mesh）导出；第二阶段在编码器输出的潜在空间上训练一个条件扩散模型（conditional diffusion model），从而能够根据文字或图像提示在数秒内生成全新的三维资产。实验表明，与同团队此前提出的显式点云生成模型 Point·E 相比，Shap·E 收敛更快、生成质量相当甚至更优，同时生成速度比基于优化的文本转3D方法快几个数量级。这项工作的意义在于，它证明了直接在高维多表示的隐式函数参数空间上做扩散生成是可行且高效的，为快速、灵活的3D内容创作提供了一条实用路径。
原文 arXiv:2305.02463;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.02463v1