HateBERT: Retraining BERT for Abusive Language Detection in English
Tommaso Caselli♣, Valerio Basile⋄, Jelena Mitrović‡, Michael Granitzer‡ ♣University of Groningen, ⋄⋄\diamondUniversity of Turin, ‡‡\ddaggerUniversity of Passau Groningen The Netherlands, Turin Italy, Passau Germany
Abstract
In this paper, we introduce HateBERT, a re-trained BERT model for abusive language detection in English. The model was trained on RAL-E, a large-scale dataset of Reddit comments in English from communities banned for being offensive, abusive, or hateful that we have collected and made available to the public. We present the results of a detailed comparison between a general pre-trained language model and the abuse-inclined version obtained by retraining with posts from the banned communities on three English datasets for offensive, abusive language and hate speech detection tasks. In all datasets, HateBERT outperforms the corresponding general BERT model. We also discuss a battery of experiments comparing the portability of the generic pre-trained language model and its corresponding abusive language-inclined counterpart across the datasets, indicating that portability is affected by compatibility of the annotated phenomena.
中文速览
网络上的仇恨言论、骂人话和冒犯性内容泛滥成灾，现有的自动检测模型大多基于通用语料训练，对这类充斥俚语和攻击性表达的特殊语言风格并不敏感。研究者从Reddit上被封禁的问题社区里收集了近150万条英文评论，构建成RAL-E数据集，并以此对通用BERT模型进行二次预训练，得到了专门面向仇恨/滥用语言的HateBERT。在冒犯性语言、滥用语言和仇恨言论三个英文基准数据集上的测试表明，HateBERT在每个任务上的宏平均F1均高于通用BERT，在滥用语言检测任务上更是刷新了当时的最优成绩。这项工作不仅验证了"用领域相关语料继续预训练"这一低成本策略的有效性，还通过跨数据集实验揭示了三类语言现象之间的specificity（具体性）差异，为内容审核研究提供了可直接使用的开源模型和数据集。
原文 arXiv:2010.12472;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2010.12472v2