Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou Center for AI Safety Carnegie Mellon University Long Phan∗ Center for AI Safety Sarah Chen∗ Center for AI Safety Stanford University James Campbell∗ Cornell University Phillip Guo∗ University of Maryland Richard Ren∗ University of Pennsylvania Alexander Pan UC Berkeley Xuwang Yin Center for AI Safety Mantas Mazeika Center for AI Safety University of Illinois Urbana-Champaign Ann-Kathrin Dombrowski Center for AI Safety Shashwat Goel Center for AI Safety Nathaniel Li Center for AI Safety UC Berkeley Michael J. Byun Stanford University Zifan Wang Center for AI Safety Alex Mallen EleutherAI Steven Basart Center for AI Safety Sanmi Koyejo Stanford University Dawn Song UC Berkeley Matt Fredrikson Carnegie Mellon University Zico Kolter Carnegie Mellon University Dan Hendrycks Center for AI Safety
Abstract
We identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems. Code is available at github.com/andyzoujm/representation-engineering.
中文速览
把深度神经网络（尤其是大语言模型）当"黑箱"对待的现状让人无法理解它们内部到底在"想"什么，这带来了严重的安全隐患。受认知神经科学中"表征空间"视角的启发，研究者提出了表征工程（Representation Engineering，RepE）这一自上而下的透明度研究框架，把模型内部的高层表征（而非单个神经元或电路）作为核心分析单位，并开发了"读取"和"控制"两类方法——前者用类似神经成像的方式定位诚实、情绪、权力欲等概念在模型中的表征位置，后者通过直接操作这些表征来改变模型行为。实验表明，RepE 方法在诚实性、无害性、幻觉检测、越狱防御等一系列安全相关任务上均有效，在 TruthfulQA 基准上以完全无监督的方式将准确率提升了 18.1 个百分点，超越了所有已有方法。这项工作证明，从表征层面切入是理解和控制大模型内在认知的一条切实可行的路径，为构建更安全、更可信的 AI 系统提供了新工具。
原文 arXiv:2310.01405;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.01405v4