Rewriting a Deep Generative Model
David Bau 11 Steven Liu 11 Tongzhou Wang 11 Jun-Yan Zhu 22 Antonio Torralba 111122
Abstract
A deep generative model such as a GAN learns to model a rich set of semantic and physical rules about the target distribution, but up to now, it has been obscure how such rules are encoded in the network, or how a rule could be changed. In this paper, we introduce a new problem setting: manipulation of specific rules encoded by a deep generative model. To address the problem, we propose a formulation in which the desired rule is changed by manipulating a layer of a deep network as a linear associative memory. We derive an algorithm for modifying one entry of the associative memory, and we demonstrate that several interesting structural rules can be located and modified within the layers of state-of-the-art generative models. We present a user interface to enable users to interactively change the rules of a generative model to achieve desired effects, and we show several proof-of-concept applications. Finally, results on multiple datasets demonstrate the advantage of our method against standard fine-tuning methods and edit transfer algorithms.
中文速览
想象一下，用生成对抗网络（GAN）造出来的图片有自己的"规则"——比如教堂楼顶永远长着尖顶、建筑物前永远空无一人——这篇论文要解决的问题就是：能不能直接修改模型内部，把这些规则换掉，而不用重新收集数据、重头训练？作者提出了"模型重写"（model rewriting）方法，把生成网络的某一卷积层看作一个线性联想记忆（linear associative memory），每个"键-值"对应一条隐含规则，然后用一个封闭形式的约束优化公式，只需用户提供一两张示例图，就能精准替换目标规则、同时尽量不破坏其余规则。实验结果表明，该方法在真实感和编辑效果上均优于标准微调方法和基于图像的编辑迁移算法，并能实现"给所有教堂楼顶换上树冠"、"批量去除水印"、"为建筑物统一添加人群"等全局性修改。这项工作的重要意义在于：它为深度生成模型提供了一把"手术刀"，让普通用户无需机器学习专业知识，就能交互式地定制和改写模型所学到的世界规则，为未来由AI生成的虚拟世界的高效创作与编辑奠定了基础。
原文 arXiv:2007.15646;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.15646v1