MotionRNN: A Flexible Model for Video Prediction with Spacetime-Varying Motions
Haixu Wu Equal contribution Zhiyu Yao Jianmin Wang Mingsheng Long (🖂) School of Software BNRist Tsinghua University China
Abstract
This paper tackles video prediction from a new dimension of predicting spacetime-varying motions that are incessantly changing across both space and time. Prior methods mainly capture the temporal state transitions but overlook the complex spatiotemporal variations of the motion itself, making them difficult to adapt to ever-changing motions. We observe that physical world motions can be decomposed into transient variation and motion trend, while the latter can be regarded as the accumulation of previous motions. Thus, simultaneously capturing the transient variation and the motion trend is the key to make spacetime-varying motions more predictable. Based on these observations, we propose the MotionRNN framework, which can capture the complex variations within motions and adapt to spacetime-varying scenarios. MotionRNN has two main contributions. The first is that we design the MotionGRU unit, which can model the transient variation and motion trend in a unified way. The second is that we apply the MotionGRU to RNN-based predictive models and indicate a new flexible video prediction architecture with a Motion Highway, which can significantly improve the ability to predict changeabl
中文速览
视频预测领域长期存在一个盲区：现有模型（如PredRNN、MIM）只会捕捉帧与帧之间的状态转移，却忽视了运动本身在时空中的持续变化，导致面对剧烈变化的运动（如雷达回波的生消、人体跑步姿态的交替）时预测能力严重下降。研究者发现，真实世界的运动可以分解为"瞬时变化（transient variation）"和"运动趋势（motion trend）"两个分量，前者描述局部区域在当前时刻的即时变化，后者反映历史运动的累积惯性。基于这一观察，他们提出了MotionRNN框架：核心部件MotionGRU用一个统一的循环单元同时建模瞬时变化和趋势动量，而Motion Highway结构则像残差连接一样将运动信息跨层直接传递，防止多层叠加时运动信号模糊乃至消失。在真实人体动作、降水临近预报和合成变速飞行数字三个具有挑战性的基准测试上，MotionRNN均取得了当时最优的预测效果，且该框架可以即插即用地嵌入现有各类RNN预测模型，具有很高的实用价值。
原文 arXiv:2103.02243;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2103.02243v3