Video Super-Resolution Transformer
Jiezhang Cao11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Yawei Li1,1{}^{1,}start_FLOATSUPERSCRIPT 1 , end_FLOATSUPERSCRIPT Kai Zhang11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPT Luc Van Gool1,212{}^{1,2}start_FLOATSUPERSCRIPT 1 , 2 end_FLOATSUPERSCRIPT 11{}^{1}start_FLOATSUPERSCRIPT 1 end_FLOATSUPERSCRIPTComputer Vision Lab, ETH Zürich, Switzerland 22{}^{2}start_FLOATSUPERSCRIPT 2 end_FLOATSUPERSCRIPTKU Leuven, Belgium {jiezhang.cao, yawei.li, kai.zhang, Co-first author.
Abstract
Video super-resolution (VSR), with the aim to restore a high-resolution video from its corresponding low-resolution version, is a spatial-temporal sequence prediction problem. Recently, Transformer has been gaining popularity due to its parallel computing ability for sequence-to-sequence modeling. Thus, it seems to be straightforward to apply the vision Transformer to solve VSR. However, the typical block design of Transformer with a fully connected self-attention layer and a token-wise feed-forward layer does not fit well for VSR due to the following two reasons. First, the fully connected self-attention layer neglects to exploit the data locality because this layer relies on linear layers to compute attention maps. Second, the token-wise feed-forward layer lacks the feature alignment which is important for VSR since this layer independently processes each of the input token embeddings without any interaction among them. In this paper, we make the first attempt to adapt Transformer for VSR. Specifically, to tackle the first issue, we present a spatial-temporal convolutional self-attention layer with a theoretical understanding to exploit the locality information. For the second is
中文速览
把视频从低分辨率还原成高分辨率（视频超分辨率，VSR）本质上是一个时空序列预测问题，而 Transformer 的并行建模能力让它看起来很适合这一任务，但标准 Transformer 里的全连接自注意力层会忽略图像的局部结构信息，逐 token 的前馈层又无法对齐不同帧之间的特征，直接套用效果很差。为此，作者提出了 VSR-Transformer，用带卷积的时空自注意力层（STCSA）替换全连接注意力，借助卷积天然捕捉局部信息，同时设计了基于双向光流的前馈层（BOFF），让不同帧的特征能够相互交互并完成对齐。在多个基准数据集上的实验表明，该方法超越了现有最先进方法，尤其在帧数较少的场景下优势更为明显，这项工作也是首次将 Transformer 系统性地适配到视频超分辨率任务的尝试，为后续相关研究提供了重要参考。
原文 arXiv:2106.06847;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2106.06847v3