Learning Depth with Convolutional Spatial Propagation Network
Xinjing Cheng, Peng Wang and Ruigang Yang X. Cheng, P. Wang, R. Yang are with Baidu Research, Baidu Inc., Beijing, China. Corresponding author: P. Wang
Abstract
In this paper, we propose the convolutional spatial propagation network (CSPN) and demonstrate its effectiveness for various depth estimation tasks. CSPN is a simple and efficient linear propagation model, where the propagation is performed with a manner of recurrent convolutional operations, in which the affinity among neighboring pixels is learned through a deep convolutional neural network (CNN). Compare to the previous state-of-the-art (SOTA) linear propagation model, i.e., spatial propagation networks (SPN), CSPN is $\bf 2$ to $\bf 5\times$ faster in practice. We concatenate CSPN and its variants to SOTA depth estimation networks, which significantly improve the depth accuracy. Specifically, we apply CSPN to two depth estimation problems: depth completion and stereo matching, in which we design modules which adapts the original 2D CSPN to embed sparse depth samples during the propagation, operate with 3D convolution and be synergistic with spatial pyramid pooling. In our experiments, we show that all these modules contribute to the final performance. For the task of depth completion, our method reduce the depth error over $\bf 30\%$ in the NYU v2 and KITTI datasets. For the ta
中文速览
深度估计中一个关键难题是：如何让神经网络预测出的深度图既边缘清晰、与图像结构对齐，又能高效完成稠密补全和立体匹配这两类任务。作者提出了卷积空间传播网络（Convolutional Spatial Propagation Network，CSPN），用循环卷积的方式让所有像素同时更新、并行传播，从数据中自动学习相邻像素之间的亲和关系，彻底改变了此前空间传播网络（SPN）必须按行或按列串行扫描的低效方式。在深度补全任务上，CSPN将稀疏激光雷达点嵌入传播过程以保留可靠测量值，在NYU v2和KITTI两个数据集上将深度误差降低超过30%；在立体匹配任务上，通过将CSPN扩展到三维视差空间，并对空间金字塔池化进行重新解读和改造，最终在KITTI 2012和2015立体匹配榜单上同时排名第一，同时实测速度比SPN快2至5倍。这项工作的意义在于，它提供了一个统一、高效且可端到端训练的传播框架，为自动驾驶和机器人所依赖的深度感知提供了切实可用的精度与速度提升。
原文 arXiv:1810.02695;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1810.02695v3