PVNAS: 3D Neural Architecture Search with Point-Voxel Convolution
Zhijian Liu*, Haotian Tang*, Shengyu Zhao, Kevin Shao, and Song Han Z. Liu, H. Tang, S. Zhao, K. Shao, and S. Han are with Department of Electrical Engineering and Computer Science, Massachusetts Institute of Technology, Cambridge, MA 02139, USA. The first two authors contributed equally to this work. E-mails:
Abstract
3D neural networks are widely used in real-world applications (e.g., AR/VR headsets, self-driving cars). They are required to be fast and accurate; however, limited hardware resources on edge devices make these requirements rather challenging. Previous work processes 3D data using either voxel-based or point-based neural networks, but both types of 3D models are not hardware-efficient due to the large memory footprint and random memory access. In this paper, we study 3D deep learning from the efficiency perspective. We first systematically analyze the bottlenecks of previous 3D methods. We then combine the best from point-based and voxel-based models together and propose a novel hardware-efficient 3D primitive, Point-Voxel Convolution (PVConv). We further enhance this primitive with the sparse convolution to make it more effective in processing large (outdoor) scenes. Based on our designed 3D primitive, we introduce 3D Neural Architecture Search (3D-NAS) to explore the best 3D network architecture given a resource constraint. We evaluate our proposed method on six representative benchmark datasets, achieving state-of-the-art performance with 1.8-23.7 $\times$ measured speedup. Furt
中文速览
自动驾驶和AR/VR设备需要对三维点云做又快又准的理解，但边缘设备算力有限，现有方案左右为难——体素化方法（voxel-based）内存消耗随分辨率立方级暴涨，而直接处理点云的方法（point-based）虽省内存却存在大量随机内存访问和动态卷积核计算，两者都难以在硬件上高效运行。为此，作者提出了点-体素卷积（Point-Voxel Convolution，PVConv）这一新型基础算子，同时保留点云的精细表达和体素化的规则内存访问优势；针对大规模室外场景，进一步引入稀疏卷积得到SPVConv；在此基础上，结合三维神经架构搜索（3D-NAS）在给定资源约束下自动找到最优网络结构。在六个主流三维理解基准上，该方法以1.8到23.7倍的实测加速同时达到了最优精度，并已在MIT无人赛车上完成实际部署，实现了更大探测范围、更低延迟和更高精度，为高效三维深度学习提供了一套完整的实用方案。
原文 arXiv:2204.11797;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2204.11797v2