Few-Shot Self-Rationalization with Natural Language Prompts
Ana Marasović Iz Beltagy Doug Downey Matthew E. Peters Allen Institute for AI, Seattle, WA, USA Equal contributions.
Abstract
Self-rationalization models that predict task labels and generate free-text elaborations for their predictions could enable more intuitive interaction with NLP systems. These models are, however, currently trained with a large amount of human-written free-text explanations for each task which hinders their broader usage. We propose to study a more realistic setting of self-rationalization using few training examples. We present FEB—a standardized collection of four existing English-language datasets and associated metrics. We identify the right prompting approach by extensively exploring natural language prompts on FEB. Then, by using this prompt and scaling the model size, we demonstrate that making progress on few-shot self-rationalization is possible. We show there is still ample room for improvement in this task: the average plausibility of generated explanations assessed by human annotators is at most 51% (with GPT-3), while plausibility of human explanations is 76%. We hope that FEB and our proposed approach will spur the community to take on the few-shot self-rationalization challenge.
中文速览
少样本情况下让模型同时预测标签、并用自然语言解释预测结果（自我理性化，self-rationalization）是当前NLP可解释性研究的痛点，因为现有方法需要大量人工撰写的解释才能训练。为此，作者构建了FEB基准——整合了四个已有英语数据集及配套评估指标——并系统探索了基于自然语言提示（prompt-based finetuning）的少样本方案，发现以问答格式设计的提示配合UnifiedQA模型效果最佳，比跨度填充提示高出8.7个准确率点。进一步扩大模型规模实验表明，GPT-3生成解释的平均可信度（plausibility）最高达51%，而人类解释的可信度为76%，说明模型与人类之间仍存在约26个百分点的差距。这项工作为社区提供了统一的评测平台与基线方法，指明了少样本自我理性化这一尚待攻克的方向。
原文 arXiv:2111.08284;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2111.08284v2