Evaluating Open-QA Evaluation
Cunxiang Wang111, Sirui Cheng, Qipeng Guo333, Yuanhao Yue444, Bowen Ding111, Zhikun Xu444, Yidong Wang111, Xiangkun Hu333, Zheng Zhang333, and Yue Zhang111 111School of Engineering, Westlake University, China 222Northeastern University, China; 333Amazon AWS AI; 444Fudan University, China {wangcunxiang, Equal contribution The corresponding author
Abstract
This study focuses on the evaluation of the Open Question Answering (Open-QA) task, which can directly estimate the factuality of large language models (LLMs). Current automatic evaluation methods have shown limitations, indicating that human evaluation still remains the most reliable approach. We introduce a new task, Evaluating QA Evaluation (QA-Eval) and the corresponding dataset EVOUNA, designed to assess the accuracy of AI-generated answers in relation to standard answers within Open-QA. Our evaluation of these methods utilizes human-annotated results to measure their performance. Specifically, the work investigates methods that show high correlation with human evaluations, deeming them more reliable. We also discuss the pitfalls of current methods and methods to improve LLM-based evaluators. We believe this new QA-Eval task and corresponding dataset EVOUNA will facilitate the development of more effective automatic evaluation tools and prove valuable for future research in this area. All resources are available at https://github.com/wangcunxiang/QA-Eval and it is under the Apache-2.0 License.
中文速览
开放域问答（Open-QA）是检验大语言模型（LLM）事实性能力的重要基准，但现有自动评估指标——尤其是精确匹配（Exact Match）——严重低估了模型的真实表现，比如同一个答案"梅西"有多种合法写法却全部被判错。为此，研究者构建了一个新的元评估任务 QA-Eval 及对应数据集 EVOUNA，用人工标注结果作为"金标准"，系统衡量词法匹配、神经网络评估（BERTScore）和基于 LLM 的自动评估器与人类判断的相关性高低。实验结果显示，三类自动方法都与人类评估存在明显差距，LLM 评估器在模型生成较长答案时尤为失准，且没有任何方法能复现人类对五个模型的相对排名。这项工作首次为 Open-QA 自动评估提供了可量化比较的基准，有助于推动更可靠的评估工具开发，对 LLM 幻觉检测研究也具有重要参考价值。
原文 arXiv:2305.12421;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2305.12421v4