Learning to Deceive with Attention-Based Explanations
Danish Pruthi†, Mansi Gupta‡, Bhuwan Dhingra†, Graham Neubig†, Zachary C. Lipton† †Carnegie Mellon University, Pittsburgh, USA ‡Twitter, New York, USA {bdhingra, gneubig,
Abstract
Attention mechanisms are ubiquitous components in neural architectures applied to natural language processing. In addition to yielding gains in predictive accuracy, attention weights are often claimed to confer interpretability, purportedly useful both for providing insights to practitioners and for explaining why a model makes its decisions to stakeholders. We call the latter use of attention mechanisms into question by demonstrating a simple method for training models to produce deceptive attention masks. Our method diminishes the total weight assigned to designated impermissible tokens, even when the models can be shown to nevertheless rely on these features to drive predictions. Across multiple models and tasks, our approach manipulates attention weights while paying surprisingly little cost in accuracy. Through a human study, we show that our manipulated attention-based explanations deceive people into thinking that predictions from a model biased against gender minorities do not rely on the gender. Consequently, our results cast doubt on attention’s reliability as a tool for auditing algorithms in the context of fairness and accountability111The code and the datasets used in
中文速览
注意力机制（attention mechanism）被广泛用于自然语言处理模型，人们普遍认为它的权重分布不仅能提升预测性能，还能"解释"模型为什么做出某个决定——比如通过观察模型是否关注性别词来判断它是否存在性别偏见。这篇论文质疑了这种"注意力即解释"的可靠性：研究者提出了一种简单的训练方法，在模型损失函数中加入惩罚项，强制让注意力权重远离指定的"不可见"词（如性别代词），同时模型实际上仍在依赖这些词进行预测。实验覆盖多个分类和序列到序列任务以及 BERT 等多种模型，结果表明这种操控几乎不损失预测精度，却能生成表面上看起来"干净"的注意力可视化图；进一步的用户研究证实，普通人在看到被操控的注意力解释后，确实会误以为模型的判断与性别无关，尽管该模型存在明显的性别偏见。这项工作揭示了一个严重的安全隐患：恶意开发者可以利用注意力可视化伪造合规性证明，误导监管机构，因此在算法公平性与问责场景中，不应将注意力权重视为可信赖的审查工具。
原文 arXiv:1909.07913;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1909.07913v2