Stochastic Image-to-Video Synthesis using cINNs
Michael Dorkenwald1 Timo Milbich1 Andreas Blattmann1 Robin Rombach1 Konstantinos G. Derpanis2,3,4 Björn Ommer1111 1IWR/HCI, Heidelberg University, Germany 2Department of Computer Science, Ryerson University, Canada 3Vector Institute for AI, Canada 4Samsung AI Centre Toronto, Canada Indicates equal supervision.
Abstract
Video understanding calls for a model to learn the characteristic interplay between static scene content and its dynamics: Given an image, the model must be able to predict a future progression of the portrayed scene and, conversely, a video should be explained in terms of its static image content and all the remaining characteristics not present in the initial frame. This naturally suggests a bijective mapping between the video domain and the static content as well as residual information. In contrast to common stochastic image-to-video synthesis, such a model does not merely generate arbitrary videos progressing the initial image. Given this image, it rather provides a one-to-one mapping between the residual vectors and the video with stochastic outcomes when sampling. The approach is naturally implemented using a conditional invertible neural network (cINN) that can explain videos by independently modelling static and other video characteristics, thus laying the basis for controlled video synthesis. Experiments on four diverse video datasets demonstrate the effectiveness of our approach in terms of both the quality and diversity of the synthesized results. Our project page is av
中文速览
视频预测模型需要理解"静态画面内容"与"动态变化信息"之间的关系，但现有方法往往只能随机生成看起来合理的续集视频，无法在初始帧与视频之间建立严格的一一对应关系。这篇论文将图像到视频的生成任务重新定义为一个可逆的域迁移问题：用一个条件可逆神经网络（conditional invertible neural network, cINN）学习初始帧与视频之间的双射映射，同时引入一个专门的"残差表示"来捕捉视频中超出初始帧的所有额外信息（如场景动态），从而让每一个残差向量与一段视频严格对应。这样，模型既能从初始帧出发采样残差、合成多样化的未来视频，也能反过来从一段视频中提取残差、解释其动态成因，还可以通过直接操控残差来控制运动方向等生成因素。在四个风格各异的视频数据集上的实验表明，该方法在生成质量和多样性上均表现出色，为可控视频生成提供了一套原理清晰、可解释的新框架。
原文 arXiv:2105.04551;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2105.04551v2