LGM: Large Multi-View Gaussian Model for High-Resolution 3D Content Creation
Jiaxiang Tang Work done while visiting S-Lab, Nanyang Technological University.11 Zhaoxi Chen 22 Xiaokang Chen 11 Tengfei Wang 33 Gang Zeng 11 Ziwei Liu 22
Abstract
3D content creation has achieved significant progress in terms of both quality and speed. Although current feed-forward models can produce 3D objects in seconds, their resolution is constrained by the intensive computation required during training. In this paper, we introduce Large Multi-View Gaussian Model (LGM), a novel framework designed to generate high-resolution 3D models from text prompts or single-view images. Our key insights are two-fold: 1) 3D Representation: We propose multi-view Gaussian features as an efficient yet powerful representation, which can then be fused together for differentiable rendering. 2) 3D Backbone: We present an asymmetric U-Net as a high-throughput backbone operating on multi-view images, which can be produced from text or single-view image input by leveraging multi-view diffusion models. Extensive experiments demonstrate the high fidelity and efficiency of our approach. Notably, we maintain the fast speed to generate 3D objects within 5 seconds while boosting the training resolution to 512, thereby achieving high-resolution 3D content generation.
中文速览
针对从文字或单张图片生成高分辨率3D模型这一难题，研究者提出了一个名为LGM（Large Multi-View Gaussian Model）的新框架：先用多视角扩散模型（multi-view diffusion model）把输入转换成四个正交视角的图像，再用一个非对称U-Net主干网络把每张图的每个像素直接预测为一个3D高斯球（3D Gaussian），将四组高斯球融合后即可通过可微渲染（differentiable rendering）进行端到端训练，最终也可导出带纹理的多边形网格。实验表明，LGM能在约5秒内生成最多65,536个高斯球组成的3D模型，训练分辨率达到512×512，远超此前基于三平面NeRF方法受限于32分辨率和128渲染上限的瓶颈。这项工作的意义在于，它同时突破了3D表示效率和骨干网络吞吐量两大障碍，让高保真、快速的3D内容生成真正走向实用。
原文 arXiv:2402.05054;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2402.05054v1