Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generators
Levon Khachatryan1∗ Andranik Movsisyan1∗ Vahram Tadevosyan1∗ Roberto Henschel1∗ Zhangyang Wang1,2 Shant Navasardyan1 Humphrey Shi1,3,4 1Picsart AI Resarch (PAIR) 2UT Austin 3U of Oregon 4UIUC https://github.com/Picsart-AI-Research/Text2Video-Zero
Abstract
Recent text-to-video generation approaches rely on computationally heavy training and require large-scale video datasets. In this paper, we introduce a new task of zero-shot text-to-video generation and propose a low-cost approach (without any training or optimization) by leveraging the power of existing text-to-image synthesis methods (e.g. Stable Diffusion), making them suitable for the video domain. Our key modifications include (i) enriching the latent codes of the generated frames with motion dynamics to keep the global scene and the background time consistent; and (ii) reprogramming frame-level self-attention using a new cross-frame attention of each frame on the first frame, to preserve the context, appearance, and identity of the foreground object. Experiments show that this leads to low overhead, yet high-quality and remarkably consistent video generation. Moreover, our approach is not limited to text-to-video synthesis but is also applicable to other tasks such as conditional and content-specialized video generation, and Video Instruct-Pix2Pix, i.e., instruction-guided video editing. As experiments show, our method performs comparably or sometimes better than recent appro
中文速览
生成连贯视频通常需要用海量视频数据从头训练模型，成本极高，而这项工作提出了一种"零样本"（zero-shot）文本生成视频的方法，完全不需要任何额外训练或微调。核心思路是直接借用已有的文本生成图像模型（如 Stable Diffusion），通过两处轻量级改造让它天然具备视频生成能力：一是在各帧的潜在编码（latent code）中注入运动动态，使全局场景和背景在时间上保持一致；二是将每一帧的自注意力机制替换为跨帧注意力，让每帧都"参照"第一帧，从而保持前景对象的外观和身份始终连贯。实验表明，这种方法在无需视频训练数据的情况下，生成质量与专门训练过的方法相当甚至更优，且开销极低。这一成果的意义在于，它把视频生成的门槛大幅降低，让普通用户无需昂贵算力就能通过文字指令生成或编辑视频。
原文 arXiv:2303.13439;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2303.13439v1