MoCoGAN: Decomposing Motion and Content for Video Generation
Sergey Tulyakov, Snap Research Ming-Yu Liu, Xiaodong Yang, Jan Kautz NVIDIA
Abstract
Visual signals in a video can be divided into content and motion. While content specifies which objects are in the video, motion describes their dynamics. Based on this prior, we propose the Motion and Content decomposed Generative Adversarial Network (MoCoGAN) framework for video generation. The proposed framework generates a video by mapping a sequence of random vectors to a sequence of video frames. Each random vector consists of a content part and a motion part. While the content part is kept fixed, the motion part is realized as a stochastic process. To learn motion and content decomposition in an unsupervised manner, we introduce a novel adversarial learning scheme utilizing both image and video discriminators. Extensive experimental results on several challenging datasets with qualitative and quantitative comparison to the state-of-the-art approaches, verify effectiveness of the proposed framework. In addition, we show that MoCoGAN allows one to generate videos with same content but different motion as well as videos with different content and same motion.
中文速览
视频天然包含"内容"（画面里有什么）和"运动"（它们怎么动）两类信息，但以往的视频生成方法（如VGAN、TGAN）把整段视频压缩成一个统一的隐向量，既无法灵活控制内容与运动，也强制要求生成固定长度的视频。MoCoGAN把图像隐空间显式拆成内容子空间和运动子空间：内容向量在整段视频里保持不变，运动向量则由一个循环神经网络逐帧生成，再通过图像生成器把两者拼合渲染成每一帧；训练时同时引入图像判别器和视频判别器，以无监督方式迫使网络学会这种分解。在多个基准数据集上，MoCoGAN在定量指标和主观质量上均优于或持平于当时最先进的方法，并能实现"换内容保运动"或"换运动保内容"这类可控生成，为后续可控视频合成研究提供了重要基础。
原文 arXiv:1707.04993;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.04993v2