Is Your Toxicity My Toxicity? Exploring the Impact of Rater Identity on Toxicity Annotation
Nitesh Goyal 0000-0002-4666-1926 Google Research, Google111 8th AveNew YorkNYUSA11201 , Ian D. Kivlichan 0000-0003-2719-2500 Jigsaw, Google111 8th AveNew YorkNYUSA11201 , Rachel Rosen 0000-0003-2927-1245 Jigsaw, Google111 8th AveNew YorkNYUSA11201 and Lucy Vasserman 0000-0002-6938-0713 Jigsaw, Google111 8th AveNew YorkNYUSA11201
Abstract
Machine learning models are commonly used to detect toxicity in online conversations. These models are trained on datasets annotated by human raters. We explore how raters’ self-described identities impact how they annotate toxicity in online comments. We first define the concept of specialized rater pools: rater pools formed based on raters’ self-described identities, rather than at random. We formed three such rater pools for this study–specialized rater pools of raters from the U.S. who identify as African American, LGBTQ, and those who identify as neither. Each of these rater pools annotated the same set of comments, which contains many references to these identity groups. We found that rater identity is a statistically significant factor in how raters will annotate toxicity for identity-related annotations. Using preliminary content analysis, we examined the comments with the most disagreement between rater pools and found nuanced differences in the toxicity annotations. Next, we trained models on the annotations from each of the different rater pools, and compared the scores of these models on comments from several test sets. Finally, we discuss how using raters that self-ide
中文速览
网络上的有毒言论（toxic language）检测模型通常依赖大量人工标注数据，但标注者的身份背景会不会影响他们对"有毒"的判断，进而让模型产生偏见？研究团队招募了三组标注者——自我认同为非裔美国人、LGBTQ群体，以及两者都不是的普通美国人——让他们对同一批涉及这些群体的评论进行毒性标注，从而直接比较不同身份群体的打分差异。结果发现，标注者的身份认同对涉及相关群体的评论打分有统计显著的影响：非裔和LGBTQ标注者在某些被外界视为有毒、但在本群体内属于自我表达的评论上，给出了明显不同的判断。基于不同群体标注数据分别训练的模型，在多个测试集上也表现出可测量的差异。这项研究说明，让目标群体成员亲自参与数据标注，能够产生更细腻、更具包容性的训练数据，有望从源头减少内容审核模型对少数群体的系统性误判。
原文 arXiv:2205.00501;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2205.00501v1