Draft-and-Revise: Effective Image Generation with Contextual RQ-Transformer
Doyup Lee POSTECH, Kakao Brain、Chiheon Kim Kakao Brain、Saehoon Kim Kakao Brain、Minsu Cho POSTECH、Wook-Shin Han POSTECH Equal contributionCorresponding author
Abstract
Although autoregressive models have achieved promising results on image generation, their unidirectional generation process prevents the resultant images from fully reflecting global contexts. To address the issue, we propose an effective image generation framework of Draft-and-Revise with Contextual RQ-transformer to consider global contexts during the generation process. As a generalized VQ-VAE, RQ-VAE first represents a high-resolution image as a sequence of discrete code stacks. After code stacks in the sequence are randomly masked, Contextual RQ-Transformer is trained to infill the masked code stacks based on the unmasked contexts of the image. Then, Contextual RQ-Transformer uses our two-phase decoding, Draft-and-Revise, and generates an image, while exploiting the global contexts of the image during the generation process. Specifically. in the draft phase, our model first focuses on generating diverse images despite rather low quality. Then, in the revise phase, the model iteratively improves the quality of images, while preserving the global contexts of generated images. In experiments, our method achieves state-of-the-art results on conditional image generation. We also va
中文速览
单向自回归模型生成图像时只能从左到右"盲猜"后续内容，无法参考整张图的全局语境，导致生成结果前后不连贯。为此，研究者提出了"草稿-修改"（Draft-and-Revise）框架：先用一种基于双向注意力的上下文感知模型（Contextual RQ-Transformer）学会在已知周围内容的情况下填补任意被遮盖的图像区域，再设计两阶段解码策略——草稿阶段从空白出发快速生成内容多样但质量较粗糙的初稿，修改阶段在保留全局结构的同时反复优化视觉细节。实验表明，该方法在类别条件和文本条件图像生成基准上达到了当时最优水平，在图像补全任务上也取得了显著提升。这项工作的价值在于，它以直觉上类似"人类先打草稿再精修"的方式突破了自回归模型的固有局限，同时提供了一个可灵活调节生成质量与多样性权衡的统一框架。
原文 arXiv:2206.04452;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2206.04452v1