Automatic Differentiation of Algorithms for Machine Learning
\NameAtılım Güneş Baydin \NameBarak A. Pearlmutter \addrDepartment of Computer Science、Hamilton Institute National University of Ireland Maynooth Co. Kildare Ireland
Abstract
Automatic differentiation—the mechanical transformation of numeric computer programs to calculate derivatives efficiently and accurately—dates to the origin of the computer age. Reverse mode automatic differentiation both antedates and generalizes the method of backwards propagation of errors used in machine learning. Despite this, practitioners in a variety of fields, including machine learning, have been little influenced by automatic differentiation, and make scant use of available tools. Here we review the technique of automatic differentiation, describe its two main modes, and explain how it can benefit machine learning practitioners. To reach the widest possible audience our treatment assumes only elementary differential calculus, and does not assume any knowledge of linear algebra.
中文速览
自动微分（Automatic Differentiation，AD）在机器学习领域长期被忽视，而研究者们却耗费大量精力手推梯度公式，或依赖存在精度问题的数值差分法。这篇文章系统介绍了 AD 的基本原理：它既不是数值微分，也不是符号微分，而是在每个基本运算上机械地应用链式法则，从而以极小的额外开销获得精确导数。文章重点讲解了前向模式（Forward Mode）和反向模式（Reverse Mode）两种 AD 变体——机器学习中最熟悉的反向传播算法本质上就是反向模式 AD 的一个特例——并梳理了 AD 在梯度优化、Hessian 计算、神经网络训练、计算机视觉及超参数调优等多个方向的实际应用价值。作者希望借此让机器学习从业者意识到：AD 工具已经成熟可用，能直接作用于含分支和循环的任意代码，若加以善用，将大幅降低开发新模型时手推导数的负担并提升计算效率。
原文 arXiv:1404.7456;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1404.7456v1