Constitutional AI: Harmlessness from AI Feedback

Anthropic's earlier RLHF models were trained to be helpful and harmless with human preference labels. They worked, but became evasive: asked something sensitive, they would refuse with "I can't help with that" instead of engaging and explaining. And the objective lived in tens of thousands of opaque human judgments.

Constitutional AI (CAI) puts the objective in writing. A constitution of 16 short principles, such as "Choose the response that a wise, ethical, polite and friendly person would more likely say", guides two stages.

In the supervised stage, a helpful-only RLHF model answers red-team prompts, which are written to provoke harmful answers. The model is then asked to critique its own answer against a randomly chosen principle and to revise it, up to four times. A pretrained model is fine-tuned on the revised answers plus ordinary helpful answers, producing SL-CAI.

In the RL stage, SL-CAI produces two answers to each red-team prompt, and a feedback model is asked, as a multiple-choice question, which answer better follows a principle. Its normalized probabilities become soft preference labels. These 182,831 AI labels for harmlessness are mixed with 135,296 human labels for helpfulness to train a preference model, and SL-CAI is trained with reinforcement learning against it. This is RL from AI feedback, or RLAIF. A variant asks the feedback model to reason step by step first.

Crowdworkers compared models in open conversation, producing Elo scores. RL-CAI models were more harmless than RLHF models trained on human harm labels at similar helpfulness, and almost never evasive. Over-training produced boilerplate and preachiness, which the authors addressed by rewording principles and clamping chain-of-thought labels.

Write the behavior policy down. For a customer's assistant, a short list of principles that a model can apply is easier to review, change and audit than a pile of labels. The same idea underlies model-judge rubrics and guardrail prompts.

AI feedback can replace much human labeling, with checks. The paper used zero human harm labels. In customer work, a model judge applying written principles can label at scale, but measure its agreement with human reviewers on a sample first; the paper did this on a 438-question evaluation.

Keep a human signal where it matters. CAI still used 135,296 human helpfulness labels. Decide which dimensions a model can judge reliably and which need people.

Define what "harmless" should look like. The authors told evaluators to prefer engaged, explained refusals over evasive ones. That choice shaped the results. Write the equivalent for a customer: when should the assistant decline, and how?

Expect over-optimization. Trained too long, RL-CAI started appending reassurance like "you are valid, valued, and cared for" to many answers. Any model trained against a model-made reward will find what that reward likes.