Reframing Human-AI Collaboration for Generating Free-Text Explanations
Sarah Wiegreffe♣♣\clubsuit Jack Hessel††\dagger Swabha Swayamdipta††\dagger Mark Riedl♣♣\clubsuit Yejin Choi◇†\Diamond\dagger ♣♣\clubsuitSchool of Interactive Computing, Georgia Institute of Technology ††\daggerAllen Institute for Artificial Intelligence ◇◇\DiamondPaul G. Allen School of Computer Science and Engineering, University of Washington
Abstract
Large language models are increasingly capable of generating fluent-appearing text with relatively little task-specific supervision. But can these models accurately explain classification decisions? We consider the task of generating free-text explanations using human-written examples in a few-shot manner. We find that (1) authoring higher quality prompts results in higher quality generations; and (2) surprisingly, in a head-to-head comparison, crowdworkers often prefer explanations generated by GPT-3 to crowdsourced explanations in existing datasets. Our human studies also show, however, that while models often produce factual, grammatical, and sufficient explanations, they have room to improve along axes such as providing novel information and supporting the label. We create a pipeline that combines GPT-3 with a supervised filter that incorporates binary acceptability judgments from humans in the loop. Despite the intrinsic subjectivity of acceptability judgments, we demonstrate that acceptability is partially correlated with various fine-grained attributes of explanations. Our approach is able to consistently filter GPT-3-generated explanations deemed acceptable by humans.
中文速览
大型语言模型（LLM）能否为分类决策生成可靠的自然语言解释，这一问题此前缺乏系统研究。研究者以GPT-3为核心，采用少样本提示（few-shot prompting）的方式，在无需大规模标注的前提下为常识问答和自然语言推理任务生成自由文本解释，并通过多轮人工评估发现：GPT-3生成的解释在语法性、事实性等表面维度表现突出，众包工人甚至常常更偏好它而非现有数据集中人工撰写的解释，但在"提供新信息"和"支持标签"等深层维度仍有明显差距。为此，研究者设计了一套"过量生成加过滤"（overgenerate-and-filter）流水线：让GPT-3反复采样多个候选解释，再用少量人工二元可接受性标注训练一个监督过滤模型，从候选中自动筛选出高质量解释。实验表明该流水线能显著提升解释的整体可接受性及深层质量，为以人工标注"哪条解释更好"替代"人工撰写解释"提供了一条更省力、更可扩展的新路径。
原文 arXiv:2112.08674;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2112.08674v2