Text2Tex: Text-driven Texture Synthesis via Diffusion Models
Dave Zhenyu Chen1 Yawar Siddiqui1 Hsin-Ying Lee2 Sergey Tulyakov2 Matthias Nießner1 1Technical University of Munich 2Snap Research https://daveredrum.github.io/Text2Tex/
Abstract
We present Text2Tex, a novel method for generating high-quality textures for 3D meshes from the given text prompts. Our method incorporates inpainting into a pre-trained depth-aware image diffusion model to progressively synthesize high resolution partial textures from multiple viewpoints. To avoid accumulating inconsistent and stretched artifacts across views, we dynamically segment the rendered view into a generation mask, which represents the generation status of each visible texel. This partitioned view representation guides the depth-aware inpainting model to generate and update partial textures for the corresponding regions. Furthermore, we propose an automatic view sequence generation scheme to determine the next best view for updating the partial texture. Extensive experiments demonstrate that our method significantly outperforms the existing text-driven approaches and GAN-based methods.
中文速览
传统3D内容制作需要设计师手工绘制贴图，耗时费力，而现有的文字驱动贴图方法又容易在不同视角之间产生拉伸、模糊和不一致的纹理瑕疵。Text2Tex利用预训练的深度感知图像扩散模型（depth-aware diffusion model），从多个视角依次对3D网格进行局部填充（inpainting），并将生成结果反投影到UV纹理空间，实现逐步"涂色"。为了解决多视角投影带来的拉伸和不一致问题，该方法动态地将每个渲染视角划分为"新生成、更新、保留、忽略"四类区域，并为不同区域分配不同的去噪强度，从而精准控制哪里需要重新生成、哪里保持不变。此外，文章还提出了自动视角排序策略，根据每个候选视角能覆盖到的"最需更新区域"面积来动态选出下一个最佳观察角度，避免人工设定视角顺序的麻烦。在Objaverse和ShapeNet等数据集上的实验表明，Text2Tex在FID、KID及用户评测等指标上均显著优于已有文字驱动方法和基于GAN的类别专用方法，为大规模自动化3D内容生产提供了一条切实可行的路径。
原文 arXiv:2303.11396;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.11396v1