Consistent123: Improve Consistency for One Image to 3D Object Synthesis
Haohan Weng 1 Tianyu Yang 2† Jianan Wang 2 Yu Li 2 Tong Zhang 1 C. L. Philip Chen 1 Lei Zhang 2 1 South China University of Technology 2 International Digital Economy Academy Work done during the internship at IDEA †Corresponding author
Abstract
Large image diffusion models enable novel view synthesis with high quality and excellent zero-shot capability. However, such models based on image-to-image translation have no guarantee of view consistency, limiting the performance for downstream tasks like 3D reconstruction and image-to-3D generation. To empower consistency, we propose Consistent123 to synthesize novel views simultaneously by incorporating additional cross-view attention layers and the shared self-attention mechanism. The proposed attention mechanism improves the interaction across all synthesized views, as well as the alignment between the condition view and novel views. In the sampling stage, such architecture supports simultaneously generating an arbitrary number of views while training at a fixed length. We also introduce a progressive classifier-free guidance strategy to achieve the trade-off between texture and geometry for synthesized object views. Qualitative and quantitative experiments show that Consistent123 outperforms baselines in view consistency by a large margin. Furthermore, we demonstrate a significant improvement of Consistent123 on varying downstream tasks, showing its great potential in the 3D
中文速览
多视角图像生成领域长期存在一个痛点：用扩散模型（diffusion model）从单张图合成多个新视角时，各视角之间往往对不上、不一致，严重妨碍了后续三维重建和图像转3D等应用。Consistent123 针对这一问题，在去噪 U-Net 中引入两种注意力机制——跨视角注意力（cross-view attention）让所有待生成视角互相"对齐"，共享自注意力（shared self-attention）则让每个新视角直接参考输入条件图的空间细节，从而在不增加训练参数的前提下显著提升一致性。采样阶段，该模型还支持同时生成任意数量的视角（训练时仅用8个视角，推理时可扩展到64个以上），并配合渐进式无分类器引导（progressive CFG）策略，在几何形状和纹理细节之间取得平衡。在多个公开基准上的实验表明，Consistent123 在视角一致性指标上大幅领先已有方法，并在三维重建和图像转3D生成等下游任务中带来显著提升，为通用3D生成提供了一个强力的基础模型。
原文 arXiv:2310.08092;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.08092v1