ArrowGAN : Learning to Generate Videos by Learning Arrow of Time
Kibeom Hong Department of Computer Science, Yonsei University, Seoul, Korea Youngjung Uh Clova AI Research, NAVER, Korea Hyeran Byun
Abstract
Training GANs on videos is even more sophisticated than on images because videos have a distinguished dimension: time. While recent methods designed a dedicated architecture considering time, generated videos are still far from indistinguishable from real videos. In this paper, we introduce ArrowGAN framework, where the discriminators learns to classify arrow of time as an auxiliary task and the generators tries to synthesize forward-running videos. We argue that the auxiliary task should be carefully chosen regarding the target domain. In addition, we explore categorical ArrowGAN with recent techniques in conditional image generation upon ArrowGAN framework, achieving the state-of-the-art performance on categorical video generation. Our extensive experiments validate the effectiveness of arrow of time as a self-supervisory task, and demonstrate that all our components of categorical ArrowGAN lead to the improvement regarding video inception score and Fréchet video distance on three datasets: Weizmann, UCFsports, and UCF-101.
中文速览
视频生成比图像生成难得多，根本原因在于视频多了一个时间维度，而现有方法在生成逼真运动方面仍然差强人意。ArrowGAN 的核心思路是把"时间之箭"（Arrow of Time，AoT）——即人类能判断一段视频是正放还是倒放的直觉——作为辅助自监督任务嵌入判别器：把真实视频倒放就能自动获得"假样本"，无需任何人工标注，判别器在学会分辨真假的同时也学会辨别正放与倒放，进而迫使生成器产出时间方向正确、运动更自然的视频。研究者还将该框架与条件批归一化、谱归一化、投影判别器等图像生成领域的成熟技术结合，扩展出可按类别生成视频的"类别化 ArrowGAN"，在 Weizmann、UCFsports 和 UCF-101 三个数据集上的视频初始分数（IS）和弗雷歇视频距离（FVD）均达到当时的最优水平。这项工作证明了为判别器选择与目标域高度相关的自监督任务至关重要，为低成本、高质量的视频生成提供了一条可推广的新思路。
原文 arXiv:2101.03710;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2101.03710v1