Evaluating and Mitigating Discrimination in Language Model Decisions
Alex Tamkin Amanda Askell Liane Lovitt Esin Durmus Nicholas Joseph Shauna Kravec Karina Nguyen Jared Kaplan Deep Ganguli
Abstract
As language models (LMs) advance, interest is growing in applying them to high-stakes societal decisions, such as determining financing or housing eligibility. However, their potential for discrimination in such contexts raises ethical concerns, motivating the need for better methods to evaluate these risks. We present a method for proactively evaluating the potential discriminatory impact of LMs in a wide range of use cases, including hypothetical use cases where they have not yet been deployed. Specifically, we use an LM to generate a wide array of potential prompts that decision-makers may input into an LM, spanning 70 diverse decision scenarios across society, and systematically vary the demographic information in each prompt. Applying this methodology reveals patterns of both positive and negative discrimination in the Claude 2.0 model in select settings when no interventions are applied. While we do not endorse or permit the use of language models to make automated decisions for the high-risk use cases we study, we demonstrate techniques to significantly decrease both positive and negative discrimination through careful prompt engineering, providing pathways toward safer depl
中文速览
语言模型越来越可能被用于贷款审批、住房资格认定等高风险决策场景，但它是否会因申请人的年龄、种族、性别而给出带有偏见的结论，目前缺乏系统的评估手段。研究者开发了一套主动评估框架：先让语言模型自动生成涵盖70种社会决策场景的提示模板，再系统替换其中的人口统计信息（年龄、种族、性别），观察模型对不同群体给出"同意"决策的概率差异。将这套方法应用于Claude 2.0后，研究发现该模型在未加干预时确实存在差别对待——对女性、非二元性别者和非白人群体倾向于给出更有利的结果（正向歧视），而对老年人则相反（负向歧视）；通过在提示中加入"歧视违法"或要求模型主动反思偏见等提示工程手段，这种差异可以被显著压缩。这项工作为开发者和监管者在语言模型真正大规模落地之前，提供了一套可复用、可扩展的歧视风险预测与缓解路径。
原文 arXiv:2312.03689;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2312.03689v1