Not Just A Black Box: Learning Important Features Through Propagating Activation Differences
Abstract
This paper describes an older version of DeepLIFT. See https://arxiv.org/abs/1704.02685 for the new version. The purported “black box” nature of neural networks is a barrier to adoption in applications where interpretability is essential. Here we present DeepLIFT (Learning Important FeaTures), an efficient and effective method for computing importance scores in a neural network. DeepLIFT compares the activation of each neuron to its ‘reference activation’ and assigns contribution scores according to the difference. We apply DeepLIFT to models trained on natural images and genomic data, and show significant advantages over gradient-based methods.
中文速览
神经网络被批评为"黑箱"，外界很难知道它做决策时究竟看重哪些输入特征，这在医疗、基因组学等需要可解释性的领域是一大障碍。DeepLIFT（Learning Important FeaTures）的核心思路是：给每个神经元设定一个"参考激活值"（当网络接收参考输入时的激活），然后把实际激活与参考激活的差值按贡献比例反向分解到各输入特征上，从而得到每个特征的重要性分数。与以往基于梯度的方法（如显著图、反卷积网络、引导反向传播）相比，DeepLIFT不会因ReLU等激活函数在不激活时梯度为零而漏掉重要信号，同时作者还证明了层级相关传播（Layer-wise Relevance Propagation）在分段线性激活下本质上等价于"梯度×输入"，而DeepLIFT通过引入非零参考激活规避了这一局限。在Tiny ImageNet图像分类和基因组序列分析两类任务上，DeepLIFT生成的特征重要性图都比梯度类方法更准确、更清晰，这为神经网络在科学发现场景中的可信部署提供了一条可行路径。
原文 arXiv:1605.01713;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1605.01713v3