Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator
Fei Wang Purdue UniversityUSA , Daniel Zheng Purdue UniversityUSA , James Decker Purdue UniversityUSA , Xilun Wu Purdue UniversityUSA , Grégory M. Essertel Purdue UniversityUSA and Tiark Rompf Purdue UniversityUSA
Abstract
Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests crucially on gradient-descent optimization and the ability to “learn” parameters of a neural network by backpropagating observed errors. However, neural network architectures are growing increasingly sophisticated and diverse, which motivates an emerging quest for even more general forms of differentiable programming, where arbitrary parameterized computations can be trained by gradient descent. In this paper, we take a fresh look at automatic differentiation (AD) techniques, and especially aim to demystify the reverse-mode form of AD that generalizes backpropagation in neural networks.
中文速览
深度学习的反向传播本质上是一种自动微分（Automatic Differentiation, AD），但它的"反向模式"对编程语言研究者来说一直显得神秘难懂。这篇论文用程序变换的视角重新解释了自动微分：把前向模式AD理解为对ANF形式程序的符号微分，把反向模式AD理解为对CPS变换程序的符号微分，并由此揭示了反向模式AD与定界续延（delimited continuations，即shift/reset控制算符）之间的深层联系——利用这一联系，反向模式AD可以完全通过算符重载实现，无需手动维护任何辅助数据结构（如"tape"）。进一步，作者将这一思路与多阶段编程（multi-stage programming）结合，构建了名为Lantern的深度学习框架，同时兼具TensorFlow"先定义后运行"的高效计算图优化能力和PyTorch"边定义边运行"的灵活控制流表达能力，在TreeLSTM、ResNet、DeepSpeech2等真实模型的GPU基准测试中验证了其性能。这项工作不仅为微分式编程提供了清晰的理论基础，也为下一代深度学习框架的设计指明了方向。
原文 arXiv:1803.10228;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1803.10228v3