IT3D: Improved Text-to-3D Generation with Explicit View Synthesis
Yiwen Chen 1,2 Chi Zhang 3 Xiaofeng Yang2 Zhongang Cai4 Gang Yu3 Lei Yang4 Guosheng Lin 1,2 1S-Lab, Nanyang Technological University 2School of Computer Science and Engineering, Nanyang Technological University 3Tencent PCG, China; 4SenseTime Research https://github.com/buaacyw/IT3D-text-to-3D Equal contribution.Corresponding Authors.
Abstract
Recent strides in Text-to-3D techniques have been propelled by distilling knowledge from powerful large text-to-image diffusion models (LDMs). Nonetheless, existing Text-to-3D approaches often grapple with challenges such as over-saturation, inadequate detailing, and unrealistic outputs. This study presents a novel strategy that leverages explicitly synthesized multi-view images to address these issues. Our approach involves the utilization of image-to-image pipelines, empowered by LDMs, to generate posed high-quality images based on the renderings of coarse 3D models. Although the generated images mostly alleviate the aforementioned issues, challenges such as view inconsistency and significant content variance persist due to the inherent generative nature of large diffusion models, posing extensive difficulties in leveraging these images effectively. To overcome this hurdle, we advocate integrating a discriminator alongside a novel Diffusion-GAN dual training strategy to guide the training of 3D models. For the incorporated discriminator, the synthesized multi-view images are considered real data, while the renderings of the optimized 3D models function as fake data. We conduct a
中文速览
现有的文本生成3D（Text-to-3D）方法依赖大型扩散模型（diffusion model）进行知识蒸馏，但常常产生过度饱和、细节不足、不真实的3D输出。本文提出的IT3D方法，先用图像到图像（image-to-image）生成管线对粗糙3D模型的渲染图进行增强，得到一批高质量的多视角图像作为参考数据集，从而绕开扩散蒸馏本身的质量瓶颈。然而，扩散模型的随机性不可避免地导致这些多视角图像之间存在内容不一致和视角不一致的问题，直接用L2监督会使3D模型向模糊的平均结果收敛。为此，文章引入一个判别器，将增强后的多视角图像视为"真实数据"、3D模型的渲染视为"假数据"，并创新性地结合扩散先验与GAN对抗损失的双训练策略（Diffusion-GAN dual training），让判别器负责把控整体分布方向、扩散先验负责补充几何和纹理细节，有效解决了小数据集导致判别器过拟合的难题。大量定性和定量实验表明，IT3D在纹理细节、几何准确性以及文本-3D语义一致性上均显著优于基线方法，同时还能加速训练收敛，并支持通过文本对已有3D模型进行外观编辑，具有较强的实用价值。
原文 arXiv:2308.11473;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2308.11473v1