Analog Bits: Generating Discrete Data using Diffusion Models with Self-Conditioning
Ting Chen, Ruixiang Zhang†, Geoffrey Hinton Google Research, Brain Team †Work done as a student researcher at Google.Code at https://github.com/google-research/pix2seq.
Abstract
We present Bit Diffusion: a simple and generic approach for generating discrete data with continuous state and continuous time diffusion models. The main idea behind our approach is to first represent the discrete data as binary bits, and then train a continuous diffusion model to model these bits as real numbers which we call analog bits. To generate samples, the model first generates the analog bits, which are then thresholded to obtain the bits that represent the discrete variables. We further propose two simple techniques, namely Self-Conditioning and Asymmetric Time Intervals, which lead to a significant improvement in sample quality. Despite its simplicity, the proposed approach can achieve strong performance in both discrete image generation and image captioning tasks. For discrete/categorical image generation, we significantly improve previous state-of-the-art on both Cifar-10 (which has $3K$ discrete 8-bit tokens) and ImageNet 64 $\times$ 64 (which has $12K$ discrete 8-bit tokens), outperforming the best autoregressive model in both sample quality (measured by FID) and efficiency. For image captioning on MS-COCO dataset, our approach achieves competitive results compared t
中文速览
离散数据（比如图像的像素值或文字的词元）用连续扩散模型来生成一直是个难题，因为扩散模型天生处理的是连续空间。这篇论文提出了"Bit Diffusion"方法：先把离散变量编码成二进制比特，再把这些0/1比特当成真实数值（称为"模拟比特"，analog bits）直接交给连续扩散模型建模，生成结束后只需做一次简单的阈值化操作就能还原出离散变量。在此基础上，作者还引入了"自条件化"（Self-Conditioning，让模型在每步去噪时参考自己上一步的预测）和"非对称时间区间"（Asymmetric Time Intervals，在采样时对时间步做细微调整）两个小技巧，显著提升了生成质量。实验表明，该方法在离散图像生成（Cifar-10、ImageNet 64×64）上大幅超越了此前最优的自回归模型，FID分数更好、推理步数更少、模型参数量更小，在MS-COCO图像描述任务上也达到了与自回归方法相当的水平，说明用连续扩散模型处理离散数据是一条简单而有效的新路子。
原文 arXiv:2208.04202;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2208.04202v2