Representation Engineering: A Top-Down Approach to AI Transparency
Andy Zou Affiliation: Center for AI Safety Affiliation: Carnegie Mellon University Long Phan∗ Affiliation: Center for AI Safety Sarah Chen∗ Affiliation: Center for AI Safety Affiliation: Stanford University James Campbell∗ Affiliation: Cornell University Phillip Guo∗ Affiliation: University of Maryland Richard Ren∗ Affiliation: University of Pennsylvania Alexander Pan Affiliation: UC Berkeley Xuwang Yin Affiliation: Center for AI Safety Mantas Mazeika Affiliation: Center for AI Safety Affiliation: University of Illinois Urbana-Champaign Ann-Kathrin Dombrowski Affiliation: Center for AI Safety Shashwat Goel Affiliation: Center for AI Safety Nathaniel Li Affiliation: Center for AI Safety Affiliation: UC Berkeley Michael J. Byun Affiliation: Stanford University Zifan Wang Affiliation: Center for AI Safety Alex Mallen Affiliation: EleutherAI Steven Basart Affiliation: Center for AI Safety Sanmi Koyejo Affiliation: Stanford University Dawn Song Affiliation: UC Berkeley Matt Fredrikson Affiliation: Carnegie Mellon University Zico Kolter Affiliation: Carnegie Mellon University Dan Hendrycks Affiliation: Center for AI Safety
Abstract
We identify and characterize the emerging area of representation engineering (RepE), an approach to enhancing the transparency of AI systems that draws on insights from cognitive neuroscience. RepE places representations, rather than neurons or circuits, at the center of analysis, equipping us with novel methods for monitoring and manipulating high-level cognitive phenomena in deep neural networks (DNNs). We provide baselines and an initial analysis of RepE techniques, showing that they offer simple yet effective solutions for improving our understanding and control of large language models. We showcase how these methods can provide traction on a wide range of safety-relevant problems, including honesty, harmlessness, power-seeking, and more, demonstrating the promise of top-down transparency research. We hope that this work catalyzes further exploration of RepE and fosters advancements in the transparency and safety of AI systems. Code is available at github.com/andyzoujm/representation-engineering.
原文 arXiv:2310.01405;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2310.01405v4