Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-Scale Convolutional Architecture
David Eigen1 Rob Fergus1,2 1 Dept. of Computer Science, Courant Institute, New York University 2 Facebook AI Research
Abstract
In this paper we address three different computer vision tasks using a single multiscale convolutional network architecture: depth prediction, surface normal estimation, and semantic labeling. The network that we develop is able to adapt naturally to each task using only small modifications, regressing from the input image to the output map directly. Our method progressively refines predictions using a sequence of scales, and captures many image details without any superpixels or low-level segmentation. We achieve state-of-the-art performance on benchmarks for all three tasks.
中文速览
一套多尺度卷积网络同时搞定深度预测、表面法线估计和语义分割这三件事——这正是本文要做的。研究者设计了一个三阶段递进式架构：第一阶段用全图视野的全连接层捕捉整体场景结构，第二阶段融合局部细节做中等分辨率预测，第三阶段进一步细化到输入图像一半分辨率的精细输出，三个任务只需改动损失函数和输出通道数便可复用同一套网络。在 NYU Depth、KITTI 等主流基准上，该方法在所有三项任务中均达到当时最优水平，且推理速度可达约 30Hz 实时运行。这项工作的意义在于证明了"一个通用像素级回归器"的可行性——过去三类任务分别依赖超像素、手工特征和繁琐后处理，而这里一个网络框架就能统一替代，为机器人、增强现实等需要多模态场景理解的系统提供了更简洁高效的基础模型。
原文 arXiv:1411.4734;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1411.4734v4