Deriving Differential Target Propagation from Iterating Approximate Inverses
Yoshua Bengio Mila、U. Montreal, CIFAR
Abstract
We show that a particular form of target propagation, i.e., relying on learned inverses of each layer, which is differential, i.e., where the target is a small perturbation of the forward propagation, gives rise to an update rule which corresponds to an approximate Gauss-Newton gradient-based optimization, without requiring the manipulation or inversion of large matrices. What is interesting is that this is more biologically plausible than back-propagation yet may turn out to implicitly provide a stronger optimization procedure. Extending difference target propagation, we consider several iterative calculations based on local auto-encoders at each layer in order to achieve more precise inversions for more accurate target propagation and we show that these iterative procedures converge exponentially fast if the auto-encoding function minus the identity function has a Lipschitz constant smaller than one, i.e., the auto-encoder is coarsely succeeding at performing an inversion. We also propose a way to normalize the changes at each layer to take into account the relative influence of each layer on the output, so that larger weight changes are done on more influential layers, like woul
中文速览
反向传播（backpropagation）虽然训练神经网络效果出色，但它要求前向和反向路径共享完全相同的权重矩阵，这在生物神经系统中几乎不可能实现。本文对一类叫做"差分目标传播"（Difference Target Propagation，DTP）的方法进行了深入的理论分析和改进：核心思路是在每一层训练一个局部自编码器来近似该层的逆映射，再把输出层的目标信号一层层"倒推"回去，为每一层提供一个只比前向激活略有偏移的局部目标，从而完成参数更新，整个过程无需跨层传递梯度。理论推导表明，当这种微小扰动的极限趋于零时，该更新规则等价于近似高斯-牛顿（Gauss-Newton）优化——这比普通随机梯度下降具有更强的二阶优化性质，却完全不需要计算或求逆大矩阵。此外，文章还证明了基于局部自编码器的迭代求逆过程在自编码器满足一个Lipschitz条件时以指数速度收敛，并提出了一种按各层对输出影响大小来归一化权重更新幅度的方案；这项工作的意义在于，它为生物合理的学习算法提供了严格的数学基础，同时表明大脑中可能存在的目标传播机制不仅在生物层面更合理，在优化层面也可能不弱于标准反向传播。
原文 arXiv:2007.15139;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2007.15139v2