StereoSet: Measuring stereotypical bias in pretrained language models
Moin Nadeem§ Work completed in part during an internship at Intel AI. Anna Bethke† Siva Reddy‡ §Massachusetts Institute of Technology Cambridge MA USA †Intel AI Santa Clara CA USA ‡Facebook CIFAR AI Chair Mila; McGill University Montreal QC Canada
Abstract
A stereotype is an over-generalized belief about a particular group of people, e.g., Asians are good at math or Asians are bad drivers. Such beliefs (biases) are known to hurt target groups. Since pretrained language models are trained on large real world data, they are known to capture stereotypical biases. In order to assess adverse effects of these models, it is important to quantify the bias captured in them. Existing literature on quantifying bias evaluates pretrained language models on a small set of artificially constructed bias-assessing sentences. We present StereoSet, a large-scale natural dataset in English to measure stereotypical biases in four domains: gender, profession, race, and religion. We evaluate popular models like BERT, GPT2, RoBERTa, and XLNet on our dataset and show that these models exhibit strong stereotypical biases. We also present a leaderboard with a hidden test set to track the bias of future language models at https://stereoset.mit.edu.
中文速览
大规模刻板印象数据集StereoSet的构建与测试，正是为了解决现有偏见评估方法依赖人工构造句子、覆盖面窄的问题。研究者通过众包平台招募美国标注者，针对性别、职业、种族、宗教四个领域收集了涵盖321个目标词、近17000条实例的自然语言测试集，每条实例都包含刻板化、反刻板化和无关三种关联选项，既能检测模型的语言建模能力，也能量化其刻板偏见程度。在此基础上，研究者设计了句内与句间两种语境关联测试（Context Association Test, CAT），分别从句子级和篇章级衡量BERT、GPT-2、RoBERTa、XLNet等主流预训练语言模型的偏见，结果发现这些模型普遍存在显著的刻板偏见倾向。这项工作的重要性在于提供了一个更贴近真实语言使用场景的偏见评测基准，并配套上线了公开排行榜，为未来持续追踪和比较各类语言模型的公平性问题提供了系统性工具。
原文 arXiv:2004.09456;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2004.09456v1