Constitutional AI: Harmlessness from AI Feedback
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion,、Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Correspondence to: Author contributions are detailed in 7. Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, Jared Kaplan††\AND Anthropic
Abstract
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as ‘Constitutional AI’. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use ‘RL from AI Feedback’ (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision
中文速览
让AI来监督AI、从而减少人工标注的需求，这是本文要解决的核心问题。研究者提出了一种叫做"宪法AI"（Constitutional AI，CAI）的训练方法：先给模型一份写有简明原则的"宪法"，让模型对自己的有害回答进行自我批评和修改，再用监督学习微调；接着用AI而非人类打分来构建偏好数据集，并以此做强化学习（即RLAIF，RL from AI Feedback）。实验结果表明，这样训练出来的助手不仅比此前依赖大量人工有害标注训练的版本更无害，还克服了以往模型遇到敏感问题就一味回避的毛病——它会主动解释自己为何拒绝，而不是沉默敷衍。这项工作的重要性在于，它证明只需一份简短的自然语言原则清单就能精确控制AI行为，大幅降低对人工标注的依赖，为未来在AI能力持续增强的背景下实现可扩展的人类监督提供了一条可行路径。
原文 arXiv:2212.08073;中英对照 + 大白话阅读 https://aha.fim.ai/paper/2212.08073v1