IPDreamer: Appearance-Controllable 3D Object Generation with Complex Image Prompts
Bohan Zeng1, Shanglin Li, Yutang Feng, Ling Yang5†, Hong Li1 Sicheng Gao1, Jiaming Liu3, Conghui He7, Wentao Zhang5, Jianzhuang Liu2 Baochang Zhang1,4, Shuicheng Yan6 1Institute of Artificial Intelligence, Beihang University 2Shenzhen Institute of Advanced Technology, Shenzhen, China 3Tiamat AI 4Zhongguancun Laboratory, Beijing, China 5Peking University 6Skywork AI 7Shanghai Artificial Intelligence Laboratory https://github.com/zengbohan0217/IPDreamer These authors contributed equally.Corresponding Author:
Abstract
Recent advances in 3D generation have been remarkable, with methods such as DreamFusion leveraging large-scale text-to-image diffusion-based models to guide 3D object generation. These methods enable the synthesis of detailed and photorealistic textured objects. However, the appearance of 3D objects produced by such text-to-3D models is often unpredictable, and it is hard for single-image-to-3D methods to deal with images lacking a clear subject, complicating the generation of appearance-controllable 3D objects from complex images. To address these challenges, we present IPDreamer, a novel method that captures intricate appearance features from complex Image Prompts and aligns the synthesized 3D object with these extracted features, enabling high-fidelity, appearance-controllable 3D object generation. Our experiments demonstrate that IPDreamer consistently generates high-quality 3D objects that align with both the textual and complex image prompts, highlighting its promising capability in appearance-controlled, complex 3D object generation.
中文速览
从复杂图片生成可控3D物体，一直是现有方法难以攻克的痛点——文本驱动的3D生成外观不可预测，而单图转3D又只能处理主体清晰的简单图片。IPDreamer提出了一套新框架，核心思路是引入"图像提示评分蒸馏采样"（Image Prompt Score Distillation Sampling, IPSDS），将复杂参考图像的细节外观特征（包括法线图）直接注入3D网格的几何与纹理优化过程，从而让生成结果的外观与参考图高度对齐。面对多张风格迥异的参考图或初始3D模型与参考图差异极大的困难场景，方法还设计了"掩码引导的组合对齐策略"，借助多模态大语言模型将不同图片的特征精准定位到3D物体的对应区域。实验表明，IPDreamer在生成质量和外观可控性上均优于现有最优方法，为工业设计、游戏、AR/VR等领域中从复杂真实图像快速生成高保真3D资产提供了切实可行的新路径。
原文 arXiv:2310.05375;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.05375v6