Teach Me to Explain: A Review of Datasets for Explainable Natural Language Processing
Sarah Wiegreffe School of Interactive Computing Georgia Institute of Technology \AndAna Marasović Allen Institute for AI University of Washington Equal contributions.
Abstract
Explainable Natural Language Processing (ExNLP) has increasingly focused on collecting human-annotated textual explanations. These explanations are used downstream in three ways: as data augmentation to improve performance on a predictive task, as supervision to train models to produce explanations for their predictions, and as a ground-truth to evaluate model-generated explanations. In this review, we identify 65 datasets with three predominant classes of textual explanations (highlights, free-text, and structured), organize the literature on annotating each type, identify strengths and shortcomings of existing collection methodologies, and give recommendations for collecting ExNLP datasets in the future.
中文速览
可解释自然语言处理（Explainable NLP）领域积累了越来越多带有人工标注文本解释的数据集，但这些数据集该如何收集、又有哪些隐患，至今缺乏系统梳理。这篇综述整理了65个包含文本解释的数据集，将其归纳为三类——高亮标注（highlights）、自由文本解释（free-text）和结构化解释（structured）——并深入分析了各类数据的标注方法、质量问题及建模与评估假设之间的矛盾，例如标注者往往只被要求标出"足够"的证据词，却未被要求做到"全面"覆盖，导致收集到的高亮标注不适合用来评估模型解释的忠实性。研究者还发现现有数据集在许可证披露、标注多样性和质量控制等方面存在普遍不足，并据此提出了一套面向未来的数据集构建建议，涵盖标注指令设计、质量控制流程和解释结构文档化等方面。这项工作填补了可解释AI领域"重方法、轻数据"的空白，为后续研究提供了可直接参考的规范框架和在线数据集目录。
原文 arXiv:2102.12060;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2102.12060v4