How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Masking
Nicola De Cao 1,2, Michael Schlichtkrull 1,2, Wilker Aziz 1, Ivan Titov 1,2 1University of Amsterdam, 2University of Edinburgh { nicola.decao, m.s.schlichtkrull, w.aziz } @uva.nl
Abstract
Attribution methods assess the contribution of inputs to the model prediction. One way to do so is erasure: a subset of inputs is considered irrelevant if it can be removed without affecting the prediction. Though conceptually simple, erasure’s objective is intractable and approximate search remains expensive with modern deep NLP models. Erasure is also susceptible to the hindsight bias: the fact that an input can be dropped does not mean that the model ‘knows’ it can be dropped. The resulting pruning is over-aggressive and does not reflect how the model arrives at the prediction. To deal with these challenges, we introduce Differentiable Masking. DiffMask learns to mask-out subsets of the input while maintaining differentiability. The decision to include or disregard an input token is made with a simple model based on intermediate hidden layers of the analyzed model. First, this makes the approach efficient because we predict rather than search. Second, as with probing classifiers, this reveals what the network ‘knows’ at the corresponding layers. This lets us not only plot attribution heatmaps but also analyze how decisions are formed across network layers. We use DiffMask to stu
中文速览
深度学习模型做出预测时，我们很难弄清它到底"看"了输入的哪些部分。抹除法（erasure）虽然直觉简单——把某个词删掉、预测不变就说明它不重要——但搜索所有可能的删除组合代价极高，而且会产生"事后偏见"：一个词能被删掉并不代表模型处理时真的忽略了它，导致归因结果过于激进、失真。为此，本文提出可微掩码方法 DiffMask，它训练一个轻量探测网络，直接利用被分析模型（如 BERT）各层的中间隐状态来预测哪些输入词元可以被遮掩，并通过 Hard Concrete 分布保持端到端可微性，无需穷举搜索。这样不仅效率大幅提升，还能像探针分类器（probing classifier）那样揭示模型在每一层"知道"哪些信息——让研究者观察到决策是如何随网络深度逐渐形成的。实验在情感分类和问答任务上验证了 DiffMask 比现有抹除近似方法更忠实、更稳定，为理解大型预训练语言模型的内部决策机制提供了实用工具。
原文 arXiv:2004.14992;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.14992v3