Quantifying Attention Flow in Transformers
Samira Abnar ILLC, University of Amsterdam \AndWillem Zuidema ILLC, University of Amsterdam
Abstract
In the Transformer model, “self-attention” combines information from attended embeddings into the representation of the focal embedding in the next layer. Thus, across layers of the Transformer, information originating from different tokens gets increasingly mixed. This makes attention weights unreliable as explanations probes. In this paper, we consider the problem of quantifying this flow of information through self-attention. We propose two methods for approximating the attention to input tokens given attention weights, attention rollout and attention flow, as post hoc methods when we use attention weights as the relative relevance of the input tokens. We show that these methods give complementary views on the flow of information, and compared to raw attention, both yield higher correlations with importance scores of input tokens obtained using an ablation method and input gradients.
中文速览
原始Transformer中的注意力权重（attention weights）被广泛用来解释模型决策，但随着层数加深，不同词元（token）的信息在隐层表示中越来越混杂，导致高层的注意力权重几乎失去可解释性。针对这一问题，研究者提出了两种追踪信息流动的方法：注意力回滚（attention rollout）将各层注意力矩阵递归相乘，把信息传播路径一路追溯到输入层；注意力流（attention flow）则把注意力图视为流网络，用最大流算法计算每个输入词元对高层表示的最大贡献量；两种方法还通过在注意力矩阵中叠加单位矩阵来纳入残差连接的影响。在主谓一致预测和情感分析任务上的实验表明，这两种方法与输入梯度及词元消融得到的重要性分数的相关性均显著高于原始注意力权重，其中注意力流更为稳健。这项工作为Transformer可解释性分析提供了简洁实用的工具，帮助研究者在不改变模型本身的前提下，更可靠地定位哪些输入词元真正影响了模型的预测结果。
原文 arXiv:2005.00928;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2005.00928v2