cuDNN: Efficient Primitives for Deep Learning
Sharan Chetlur, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran NVIDIA Santa Clara, CA 95050 {schetlur, jwoolley, philippev, jocohen, \AndBryan Catanzaro Baidu Research Sunnyvale, CA 94089 \AndEvan Shelhamer UC Berkeley Berkeley, CA 94720
Abstract
We present a library of efficient implementations of deep learning primitives. Deep learning workloads are computationally intensive, and optimizing their kernels is difficult and time-consuming. As parallel architectures evolve, kernels must be reoptimized, which makes maintaining codebases difficult over time. Similar issues have long been addressed in the HPC community by libraries such as the Basic Linear Algebra Subroutines (BLAS) [2]. However, there is no analogous library for deep learning. Without such a library, researchers implementing deep learning workloads on parallel processors must create and optimize their own implementations of the main computational kernels, and this work must be repeated as new parallel processors emerge. To address this problem, we have created a library similar in intent to BLAS, with optimized routines for deep learning workloads. Our implementation contains routines for GPUs, although similarly to the BLAS library, these routines could be implemented for other platforms. The library is easy to integrate into existing frameworks, and provides optimized performance and memory usage. For example, integrating cuDNN into Caffe, a popular framework
中文速览
深度学习训练时最耗时的卷积、池化等核心运算，过去只能由各框架自己手写GPU实现，每换一代硬件就得重头优化，费时费力还难以维护。为此，NVIDIA推出了cuDNN这个专为深度学习设计的底层原语库，提供卷积（含前向与反向传播）、池化、激活函数、Softmax等高度优化的GPU实现，并以简洁的C语言API接入现有框架，同时刻意将辅助内存占用压到最低——没有im2col展开带来的数据膨胀，也没有FFT所需的大块临时缓冲区。实测结果显示，把cuDNN集成进Caffe后，标准卷积网络的训练速度提升36%，内存占用同步下降。这个库的意义在于，它让框架开发者不必再纠缠硬件细节，只需调用稳定接口就能跟上GPU架构迭代的步伐，类似于线性代数领域的BLAS之于HPC社区。
原文 arXiv:1410.0759;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1410.0759v3