HateCheck: Functional Tests for Hate Speech Detection Models
Paul Röttger University of Oxford The Alan Turing Institute Bertram Vidgen The Alan Turing Institute Dong Nguyen Utrecht University Zeerak Waseem University of Sheffield Helen Margetts University of Oxford The Alan Turing Institute Janet B. Pierrehumbert University of Oxford
Abstract
Detecting online hate is a difficult task that even state-of-the-art models struggle with. Typically, hate speech detection models are evaluated by measuring their performance on held-out test data using metrics such as accuracy and F1 score. However, this approach makes it difficult to identify specific model weak points. It also risks overestimating generalisable model performance due to increasingly well-evidenced systematic gaps and biases in hate speech datasets. To enable more targeted diagnostic insights, we introduce HateCheck, a suite of functional tests for hate speech detection models. We specify 29 model functionalities motivated by a review of previous research and a series of interviews with civil society stakeholders. We craft test cases for each functionality and validate their quality through a structured annotation process. To illustrate HateCheck’s utility, we test near-state-of-the-art transformer models as well as two popular commercial models, revealing critical model weaknesses.
中文速览
网上仇恨言论检测模型长期存在一个痛点：现有评估方式只看模型在测试集上的整体准确率，既看不出具体短板在哪里，又因为数据集本身的系统性偏差而高估了模型的真实能力。为此，研究者推出了 HateCheck——一套针对仇恨言论检测模型的功能测试（functional testing）框架，通过回顾大量文献并访谈21位来自民间社会组织的实践人员，归纳出29项模型应具备的功能，再为每项功能精心手工构造带有明确标准答案的测试用例，共生成3901条，并经过严格的标注验证。用这套工具测试接近最优水平的BERT类模型和谷歌Perspective、Two Hat SiftNinja两款商用模型后，发现所有模型都过度依赖"脏话""群体标识词"等关键词，对否定句式的仇恨、反驳性言论等非仇恨表达普遍判断失误，BERT模型在不同受保护群体之间的表现也存在明显偏差。HateCheck的意义在于，它首次为研究者和开发者提供了一把可复用的"诊断仪"，能系统性地暴露模型的具体弱点，从而推动更可靠的仇恨言论检测系统的开发。
原文 arXiv:2012.15606;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2012.15606v2