ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning
Olga Golovneva, Moya Chen, Spencer Poff, Martin Corredor, Luke Zettlemoyer, Maryam Fazel-Zarandi, Asli Celikyilmaz Meta AI Research {olggol, mpchen, spoff, mcorredor, lsz, maryamfazel,
Abstract
Large language models show improved downstream task performance when prompted to generate step-by-step reasoning to justify their final answers (Nye et al., 2021; Wei et al., 2022). These reasoning steps greatly improve model interpretability and verification, but objectively studying their correctness (independent of the final answer) is difficult without reliable methods for automatic evaluation. We simply do not know how often the stated reasoning steps actually support the final end task predictions. In this work, we present ROSCOE, a suite of interpretable, unsupervised automatic scores that improve and extend previous text generation evaluation metrics. To evaluate ROSCOE against baseline metrics, we design a typology of reasoning errors and collect synthetic and human evaluation scores on commonly used reasoning datasets. In contrast with existing metrics, ROSCOE can measure semantic consistency, logicality, informativeness, fluency, and factuality — among other traits — by leveraging properties of step-by-step rationales. We empirically verify the strength of our metrics on five human annotated and six programmatically perturbed diagnostics datasets - covering a diverse set
中文速览
大型语言模型（LLM）在被要求逐步推理后，答题表现明显提升，但这些"推理链"到底写得对不对，目前缺乏可靠的自动评估手段。为此，研究者提出了 ROSCOE 这套无监督、可解释的细粒度自动评分体系，从语义对齐、语义相似度、逻辑推断和语言连贯性四个维度，对逐步推理链的每个环节分别打分，而非笼统地给整段输出一个分数。他们还专门设计了一套推理错误分类体系，并据此构建了五个人工标注数据集和六个程序化扰动数据集，覆盖算术、逻辑、常识等多类推理任务，用来对比 ROSCOE 与现有基线指标的表现。实验结果显示，ROSCOE 在所有需要基于上下文推理的任务上均持续优于基线，且对逻辑和算术推理场景表现出更强的鲁棒性。这项工作填补了"推理链质量自动评估"这一关键空白，为今后研究 LLM 推理可靠性与可解释性提供了实用工具和评估基准。
原文 arXiv:2212.07919;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.07919v2