VideoFlow: A Conditional Flow-Based Model for Stochastic Video Generation
Manoj Kumar , Mohammad Babaeizadeh, Dumitru Erhan, Chelsea Finn, Sergey Levine, Laurent Dinh, Durk Kingma Google Research, Brain Team A majority of this work was done as part of the Google AI Residency Program.
Abstract
Generative models that can model and predict sequences of future events can, in principle, learn to capture complex real-world phenomena, such as physical interactions. However, a central challenge in video prediction is that the future is highly uncertain: a sequence of past observations of events can imply many possible futures. Although a number of recent works have studied probabilistic models that can represent uncertain futures, such models are either extremely expensive computationally as in the case of pixel-level autoregressive models, or do not directly optimize the likelihood of the data. To our knowledge, our work is the first to propose multi-frame video prediction with normalizing flows, which allows for direct optimization of the data likelihood, and produces high-quality stochastic predictions. We describe an approach for modeling the latent space dynamics, and demonstrate that flow-based generative models offer a viable and competitive approach to generative modeling of video.
中文速览
视频未来帧预测的核心难题在于未来的不确定性——同一段历史画面可能对应无数种后续走向，而现有模型要么计算代价极高（逐像素自回归生成），要么无法直接优化数据的真实似然（如VAE）。研究者提出了VideoFlow，首次将基于归一化流（normalizing flows）的生成模型引入多帧视频预测任务：通过可逆变换把每一帧映射到隐空间，再用一个自回归动态模型在隐空间中建模时序依赖，从而既能精确计算数据似然，又能高效并行地采样出多样化的未来帧。在无动作标注的BAIR机器人推物数据集上，VideoFlow的定量指标与最优VAE方案相当，预测画面清晰无模糊伪影，同时比最快的自回归视频模型快数十倍。这项工作证明了归一化流可以成为随机视频预测的可行路线，为需要实时未来预测的机器人控制等应用场景提供了更实用的生成建模方案。
原文 arXiv:1903.01434;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1903.01434v3