Scaling Autoregressive Video Models
Dirk Weissenborn Google Research、Oscar Täckström∗ Sana Labs、Jakob Uszkoreit Google Research Equal contribution.Work done while at Google Research.
Abstract
Due to the statistical complexity of video, the high degree of inherent stochasticity, and the sheer amount of data, generating natural video remains a challenging task. State-of-the-art video generation models often attempt to address these issues by combining sometimes complex, usually video-specific neural network architectures, latent variable models, adversarial training and a range of other methods. Despite their often high complexity, these approaches still fall short of generating high quality video continuations outside of narrow domains and often struggle with fidelity. In contrast, we show that conceptually simple autoregressive video generation models based on a three-dimensional self-attention mechanism achieve competitive results across multiple metrics on popular benchmark datasets, for which they produce continuations of high fidelity and realism. We also present results from training our models on Kinetics, a large scale action recognition dataset comprised of YouTube videos exhibiting phenomena such as camera movement, complex object interactions and diverse human movement. While modeling these phenomena consistently remains elusive, we hope that our results, whic
中文速览
视频生成一直面临数据量庞大、随机性强、时空建模复杂等挑战，现有方法往往依赖精心设计的专用网络架构和对抗训练，却仍难以生成高质量、高保真的视频。研究者提出了一种概念上简洁的"视频Transformer"（Video Transformer），其核心是三维块局部自注意力（3D block-local self-attention）机制，将视频整体视为时空三维体积而非独立帧序列，同时结合三维时空下采样（spatiotemporal subscaling）来大幅压缩内存占用。在机器人推物（BAIR）等主流基准数据集上，该模型在困惑度等多项指标上达到甚至超过现有最优水平，并在Kinetics-600大规模动作识别数据集的烹饪视频子集上生成了具有合理物体交互和摄像机运动的视频续段。这项工作证明了无需复杂专用设计、仅凭纯自回归加自注意力的统一框架就能在视频生成领域取得竞争性成果，也为未来在更具挑战性的大规模视频数据集上开展研究提供了有力的出发点。
原文 arXiv:1906.02634;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1906.02634v3