SSR-Encoder: Encoding Selective Subject Representation for Subject-Driven Generation
Yuxuan Zhang1*, Yiren Song4, Jiaming Liu2††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT, Rui Wang3*, Jinpeng Yu6*, Hao Tang5, Huaxia Li2, Xu Tang2, Yao Hu2, Han Pan1, Zhongliang Jing1††{}^{\dagger}start_FLOATSUPERSCRIPT † end_FLOATSUPERSCRIPT
Abstract
Recent advancements in subject-driven image generation have led to zero-shot generation, yet precise selection and focus on crucial subject representations remain challenging. Addressing this, we introduce the SSR-Encoder, a novel architecture designed for selectively capturing any subject from single or multiple reference images. It responds to various query modalities including text and masks, without necessitating test-time fine-tuning. The SSR-Encoder combines a Token-to-Patch Aligner that aligns query inputs with image patches and a Detail-Preserving Subject Encoder for extracting and preserving fine features of the subjects, thereby generating subject embeddings. These embeddings, used in conjunction with original text embeddings, condition the generation process. Characterized by its model generalizability and efficiency, the SSR-Encoder adapts to a range of custom models and control modules. Enhanced by the Embedding Consistency Regularization Loss for improved training, our extensive experiments demonstrate its effectiveness in versatile and high-quality image generation, indicating its broad applicability. Project page: ssr-encoder.github.io
中文速览
主题驱动图像生成（subject-driven image generation）的核心难题在于：如何从包含背景、风格等复杂信息的参考图中，精准地"抠出"用户真正关心的那个主体。SSR-Encoder 提出了一套无需在测试阶段额外微调的编码器框架，通过"词元到图块对齐器"（Token-to-Patch Aligner）将文本或遮罩查询与图像局部区域对齐，再配合"细节保留主体编码器"（Detail-Preserving Subject Encoder）从 CLIP 视觉骨干的多个层级提取精细特征，最终生成能够精确表征目标主体的多尺度嵌入，并以此条件化扩散模型的生成过程。大量实验表明，该方法在无微调方法中达到最优水平，甚至能与需要微调的方法相媲美，同时兼容 ControlNet 等各类控制模块和自定义的 Stable Diffusion 模型。这项工作的重要性在于，它让用户只需一张参考图加上简单的文字或涂抹，就能灵活、高保真地生成任意指定主体，大幅降低了个性化图像生成的门槛。
原文 arXiv:2312.16272;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2312.16272v2