Revisiting Hierarchical Approach for Persistent Long-Term Video Prediction
Wonkwang Lee1, Whie Jung1, Han Zhang2, Ting Chen2, Jing Yu Koh2, Thomas Huang3, Hyungsuk Yoon4, Honglak Lee3,5111The author contributed to this work while at Google Research., Seunghoon Hong1 1KAIST, 2Google Research, 3University of Michigan, 4MOLOCO, 5LG AI Research {wonkwang.lee, whieya, {zhanghan, iamtingchen,
Abstract
Learning to predict the long-term future of video frames is notoriously challenging due to inherent ambiguities in the distant future and dramatic amplifications of prediction error through time. Despite the recent advances in the literature, existing approaches are limited to moderately short-term prediction (less than a few seconds), while extrapolating it to a longer future quickly leads to destruction in structure and content. In this work, we revisit hierarchical models in video prediction. Our method predicts future frames by first estimating a sequence of semantic structures and subsequently translating the structures to pixels by video-to-video translation. Despite the simplicity, we show that modeling structures and their dynamics in the discrete semantic structure space with a stochastic recurrent estimator leads to surprisingly successful long-term prediction. We evaluate our method on three challenging datasets involving car driving and human dancing, and demonstrate that it can generate complicated scene structures and motions over a very long time horizon (i.e., thousands frames), setting a new standard of video prediction with orders of magnitude longer prediction ti
中文速览
视频里预测遥远未来一直是个大难题：误差会随时间快速累积，画面很快就会糊成一团。这项工作提出一种层次化视频预测（Hierarchical Video Prediction, HVP）方法，核心思路是先让模型在"语义分割标签图"这个低维、离散的结构空间里预测未来场景布局，再用视频到视频翻译技术把预测出的标签序列渲染成真实像素；两个阶段分别用变分循环网络和条件对抗生成网络来完成。实验在驾驶场景和人体舞蹈等数据集上表明，该方法能连贯地生成长达数千帧的未来视频，预测时长比现有方法高出几个数量级，且画质不会随时间明显退化。这一成果意味着机器有望对真实世界做出更长远、更可靠的视觉预判，对自动驾驶规划和模型驱动强化学习等下游任务具有重要的实际价值。
原文 arXiv:2104.06697;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2104.06697v1