LRM: Large Reconstruction Model for Single Image to 3D
Yicong Hong1,212{}^{1,2}start_FLOATSUPERSCRIPT 1 , 2 end_FLOATSUPERSCRIPT Kai Zhang11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Jiuxiang Gu11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Sai Bi11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Yang Zhou11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Difan Liu11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Feng Liu11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Kalyan Sunkavalli11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Trung Bui11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Hao Tan11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPTAdobe Research 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTAustralian National Univeristy Intern at Adobe Research.
Abstract
We propose the first Large Reconstruction Model (LRM) that predicts the 3D model of an object from a single input image within just 5 seconds. In contrast to many previous methods that are trained on small-scale datasets such as ShapeNet in a category-specific fashion, LRM adopts a highly scalable transformer-based architecture with 500 million learnable parameters to directly predict a neural radiance field (NeRF) from the input image. We train our model in an end-to-end manner on massive multi-view data containing around 1 million objects, including both synthetic renderings from Objaverse and real captures from MVImgNet. This combination of a high-capacity model and large-scale training data empowers our model to be highly generalizable and produce high-quality 3D reconstructions from various testing inputs, including real-world in-the-wild captures and images created by generative models. Video demos and interactable 3D meshes can be found on our LRM project webpage: https://yiconghong.me/LRM.
中文速览
单张照片重建3D物体一直是计算机视觉领域的难题，传统方法要么只针对特定类别、要么需要漫长的逐形状优化。LRM（大规模重建模型）借鉴大语言模型的成功经验，搭建了一个拥有5亿参数的纯Transformer编码器-解码器架构：先用预训练的DINO视觉模型提取输入图像的特征，再通过交叉注意力机制将2D特征投影到三平面（triplane）NeRF表示上，最终以体渲染方式输出任意视角的颜色和密度。模型在约100万个三维物体的多视角数据（Objaverse合成渲染与MVImgNet真实视频）上端到端训练，无需复杂的3D正则化或精细的超参数调整。结果表明，LRM仅需5秒即可从单张图片生成高保真3D网格，对现实拍摄照片和生成模型生成的图片均有出色泛化能力，为工业设计、游戏、AR/VR等领域的快速3D内容生产提供了切实可行的新范式。
原文 arXiv:2311.04400;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2311.04400v2