MVDream: Multi-view Diffusion for 3D Generation
Yichun Shi1, Peng Wang1, Jianglong Ye2, Long Mai1, Kejie Li1, Xiao Yang1 1 ByteDance, USA, 2 University of California, San Diego Work done during internship at ByteDance
Abstract
We introduce MVDream, a diffusion model that is able to generate consistent multi-view images from a given text prompt. Learning from both 2D and 3D data, a multi-view diffusion model can achieve the generalizability of 2D diffusion models and the consistency of 3D renderings. We demonstrate that such a multi-view diffusion model is implicitly a generalizable 3D prior agnostic to 3D representations. It can be applied to 3D generation via Score Distillation Sampling, significantly enhancing the consistency and stability of existing 2D-lifting methods. It can also learn new concepts from a few 2D examples, akin to DreamBooth, but for 3D generation. Our project page is https://MV-Dream.github.io
中文速览
想让AI根据文字描述自动生成高质量3D内容，最大的拦路虎是"多面Janus问题"——扩散模型从不同角度渲染同一物体时会反复生成相似的正面，导致3D结果前后左右自相矛盾。MVDream的解决思路是训练一个多视角扩散模型（multi-view diffusion model），让它一次性同时生成同一物体的多张视角一致的图像：方法上，在Stable Diffusion的自注意力层做"3D膨胀"，把所有视角的token打通做联合注意力，并以相机外参作为额外条件注入模型，同时混合2D图文数据和3D渲染数据联合训练，从而既继承2D扩散模型的泛化能力，又学到真正的多视角几何一致性。把这个多视角先验接入分数蒸馏采样（Score Distillation Sampling，SDS）流程后，生成的NeRF模型几乎消除了Janus问题，稳定性和多样性均优于或持平于当时最先进的方法，还能像DreamBooth那样从少量2D图片微调出特定角色的3D生成能力。这项工作表明，只要让扩散模型同时"看"多个视角，就能成为通用的3D生成先验，为后续文本驱动3D内容创作提供了简洁而有效的基础框架。
原文 arXiv:2308.16512;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.16512v4