A causal framework for explaining the predictions of black-box sequence-to-sequence models
David Alvarez-Melis Tommi S. Jaakkola CSAIL, MIT {davidam,
Abstract
We interpret the predictions of any black-box structured input-structured output model around a specific input-output pair. Our method returns an “explanation” consisting of groups of input-output tokens that are causally related. These dependencies are inferred by querying the black-box model with perturbed inputs, generating a graph over tokens from the responses, and solving a partitioning problem to select the most relevant components. We focus the general approach on sequence-to-sequence problems, adopting a variational autoencoder to yield meaningful input perturbations. We test our method across several NLP sequence generation tasks.
中文速览
针对神经机器翻译等复杂黑箱模型"预测了什么却说不清为什么"这一痛点，研究者提出了一套名为 SocRat 的解释框架：先用变分自编码器（variational autoencoder, VAE）对输入句子做语义保留的扰动，再把扰动后的输入-输出对喂给黑箱模型，通过贝叶斯逻辑回归推断原始输入词元与输出词元之间的因果依赖强度，最后将这些依赖关系构建成一张二部图，并用鲁棒最小割（min-cut）划分出若干"解释组块"，每个组块对应一组相互因果关联的输入-输出词元。实验表明，该方法能从零恢复字素-音素词典中的已知对应关系，并且与神经机器翻译系统内部的注意力权重高度吻合，还能揭示翻译系统中的潜在偏差。这项工作的意义在于，它是首个同时面向结构化输入和结构化输出的黑箱解释框架，无需访问模型内部参数，为提升复杂 NLP 系统的可信度和可诊断性提供了实用工具。
原文 arXiv:1707.01943;中英对照 + 大白话阅读 https://aha.fim.ai/paper/1707.01943v3