How Auto-Encoders Could Provide Credit Assignment in Deep Networks via Target Propagation
Yoshua Bengio Université de Montréal CIFAR Fellow
Abstract
We propose to exploit reconstruction as a layer-local training signal for deep learning. Reconstructions can be propagated in a form of target propagation playing a role similar to back-propagation but helping to reduce the reliance on derivatives in order to perform credit assignment across many levels of possibly strong non-linearities (which is difficult for back-propagation). A regularized auto-encoder tends produce a reconstruction that is a more likely version of its input, i.e., a small move in the direction of higher likelihood. By generalizing gradients, target propagation may also allow to train deep networks with discrete hidden units. If the auto-encoder takes both a representation of input and target (or of any side information) in input, then its reconstruction of input representation provides a target towards a representation that is more likely, conditioned on all the side information. A deep auto-encoder decoding path generalizes gradient propagation in a learned way that can could thus handle not just infinitesimal changes but larger, discrete changes, hopefully allowing credit assignment through a long chain of non-linear operations. In addition to each layer bei
中文速览
深度神经网络训练长期依赖反向传播（back-propagation），但反向传播在网络很深、存在强非线性或离散隐变量时往往面临梯度消失等信用分配难题。这篇论文提出用"目标传播"（target propagation）替代或补充反向传播：在网络每一层引入一个正则化自编码器（auto-encoder），让它的重建输出作为该层的局部训练目标，从而把"应该往哪个方向调整"这一信号一层一层地向下传递，而不再完全依赖梯度链式求导。这种做法不仅天然支持离散隐单元，还可以扩展到有监督、多模态和循环网络场景——当目标标签或其他模态信息同时输入自编码器时，重建结果便自动融合了这些约束，为下层提供更有信息量的训练目标。作者从变分自编码器（variational auto-encoder）的理论框架出发，系统梳理了该方案的数学动机与合理性，并进一步猜想这一机制在生物神经网络中可能正是大脑跨多层非线性、噪声和离散变换完成信用分配的方式，为理解深度学习与神经科学的联系提供了新视角。
原文 arXiv:1407.7906;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1407.7906v3