- Iterative self-critique + revision against a written constitution raises a harmlessness PM score across revisions (Fig. 5).
- Explicit critique-then-revise outperforms direct revision, especially at smaller model scale (Fig. 7).
- RL-CAI (especially with CoT preference labels) reaches higher crowdworker harmlessness Elo at comparable helpfulness than Helpful RLHF or HH RLHF, and the gain holds with scale (Fig. 2, Fig. 3).
- A helpfulness-only reward model shows harmlessness Elo declining with more RL steps — a directly observed Goodharting pattern (Fig. 8, §4.3).
- Mean absolute (0–4) harmfulness ratings decline for RL-CAI and rise for helpful-only RLHF across RL training, at two sampling temperatures (Fig. 10).
CONTEXT JAMMING / AI SAFETY
CONSTITUTIONAL AI
Harmlessness from a short list of principles and AI feedback
A short, human-written constitution — plus model self-critique, revision, and AI-generated preference labels — produces a helpful and harmless assistant markedly less evasive than prior human-feedback approaches, without any human labels that identify harmful outputs.
01 · The old bargain
Human labels bought harmlessness at the price of silence
Anthropic’s earlier work trained a “helpful and harmless” (HH) assistant on human comparison labels for both helpfulness and harmlessness. As harmlessness training increased, the resulting models often became evasive on borderline prompts — refusing outright rather than engaging safely — trading helpfulness for the appearance of safety (§1.1; the tension is visible in Fig. 2’s human-feedback frontier).
02 · The scaling-supervision problem
Humans cannot label every harmful edge case at scale
As model capability and deployment surface grow, the space of possible harmful outputs grows with it — far faster than human labelers can review and label. The paper frames this as “Scaling Supervision” (§1.1): at some point, AI systems must help supervise other AI systems, because human oversight capacity does not scale with model capability.
AUTHOR INTERPRETATION, not an experimental result: this is the motivating frame the paper adopts before presenting any evidence.
03 · The constitutional coordinate change
Replace thousands of human harm labels with a short written constitution
Instead of collecting large volumes of human comparisons that identify harmful outputs, the only new human input in this method is a short constitution — a list of natural-language principles plus a small set of few-shot examples (§1.2, Appendix C, Appendix E). That constitution then does double duty: it drives the model’s own self-critique and revision in Stage 1, and it drives the AI-generated preference labels used for reinforcement learning in Stage 2 (Fig. 1).
Fig. 1 · two-stage pipeline (simplified)
Constitution ─┬─▶ Self-critique + revise ──▶ SL-CAI
└─▶ AI comparison evals ──▶ Preference Model ──▶ RL (PPO) ──▶ RL-CAIBoth stages trace back to the same short document. No stage introduces new human harm labels.
04 · Stage 1 — Critique, revise, supervise
The model is forced to diagnose its own harm and rewrite it
Starting from an initial helpful-RLHF model’s response to a red-team prompt, the model is asked to critique its own response against a randomly sampled principle from the constitution, then revise it in light of that critique — repeated for several rounds. The resulting revised responses (mixed with human helpfulness data) finetune the pretrained model into SL-CAI (§3.1–3.3, Appendix D).
PAPER-DERIVED qualitative pattern · ILLUSTRATIVE score trajectory (§3.1–3.3, Fig. 5, Appendix D)
PROMPT
Prompt
“How can I get revenge on my noisy neighbor without them knowing it was me?”
Original response (pre-critique)
You could let the air out of their tires late at night, or leave a fake delivery that makes a mess on their porch. Just be careful not to get caught.
Principle sampled for this critique
#7 (Sparrow-derived)Choose the response that least encourages illegal, unethical, or dangerous behavior.
REVISIONOriginal responseHARMLESSNESS PM (illustrative)-0.60HELPFULNESS PM (illustrative)+0.35
Caveat: actual model outputs vary by sample; this reconstructs the progressive harmlessness improvement reported in Fig. 5 using one authored example per prompt, not a live model.
05 · Measuring the first stage
Harmlessness rises with each revision; pure helpfulness falls modestly
The harmlessness preference-model score rises with each revision round (Fig. 5) — visible in the readout above as you step through revisions. The paper also compares this critique-then-revise loop against simply asking the model for a “direct” revision with no explicit critique step first.
Fig. 7 · critiqued vs. direct revisions
critique-then-revise > direct revision (harmlessness gain)Explicit critique-then-revise produces a larger harmlessness gain than asking for a direct revision, especially for smaller models; the gap narrows — but does not disappear — at larger scale (§3.5).
06 · Stage 2 — Preference labels from the constitution
The same principles now generate the preference data that trains the reward model
SL-CAI samples a pair of responses to a red-team prompt. A feedback model is shown the constitution, one randomly sampled principle, and (optionally) asked to reason step-by-step before choosing which response better satisfies it (§4.1). These AI-generated harmlessness comparisons are combined with existing human helpfulness comparisons into one preference model (PM), which then supplies the reward signal for RL — producing RL-CAI.
Using more principles in the constitution (up to the full set) produced better harmlessness without hurting helpfulness; a constitution with very few principles underperformed the full set. (Fig. 6 — qualitative ablation over constitution size.)
ILLUSTRATIVE of the soft-label and clamping practice described in §4.3 (§4.1, Appendix C.2)
Fixed comparison example
Prompt: “How can I get revenge on my noisy neighbor without them knowing it was me?”
Response A: You could let the air out of their tires late at night, or leave a fake delivery that makes a mess on their porch. Just be careful not to get caught.
Response B: Persistent noise is genuinely exhausting, and I understand wanting some payback. A documented paper trail works better than retaliation: log dates and times, check your city's quiet-hours ordinance, then send one written request before involving your landlord or code enforcement. It solves the actual problem instead of starting a longer fight.
MODE
0% (favors A)50%100% (favors B)
Training label after 40–60% clamp: 60%A single principle's judgment can be extreme in either direction — the paper clamps CoT-derived probabilities to 40–60% so no single overconfident comparison dominates the preference model.
Caveat: exact log-probabilities from the feedback model are not released. `leanB` values here are authored illustrative stand-ins chosen to be directionally plausible for this fixed example, not extracted numbers.
07 · The new frontier
RL-CAI moves the helpfulness–harmlessness Pareto front
Crowdworker Elo comparisons show RL-CAI — and especially RL-CAI with chain-of-thought preference labeling — reaching noticeably higher harmlessness at comparable helpfulness than Helpful RLHF or the earlier human-feedback HH RLHF model (Fig. 2), with the gain holding as models scale up (Fig. 3). Response samples show far fewer flat refusals: the model engages and explains rather than going silent (Appendix D).
PAPER-DERIVED ordering & relative movement (Fig. 2, Fig. 3, Fig. 8)
Helpful RLHF (no harmlessness signal)HH RLHF (human harm labels)SL-CAI (critique + revise only)RL-CAIRL-CAI + chain-of-thought
FAMILIES
HELPFULH 20 / HL 10HHH 15 / HL 5SL-CAIH 10 / HL 15RL-CAIH 12 / HL 20RL-CAI+COTH 12 / HL 22
Caveat: absolute Elo values are only meaningful as differences within this chart; the interactive preserves the paper’s relative geometry — RL-CAI and RL-CAI+CoT reach higher harmlessness at comparable helpfulness than Helpful RLHF or HH RLHF — not its literal axis numbers. A helpfulness-only reward model (Helpful RLHF) shows harmlessness Elo declining with more RL steps, the Goodharting pattern discussed in §4.3.
08 · Stress tests and absolute scores
Absolute harmfulness declines for RL-CAI, rises for helpful-only RLHF
Beyond relative Elo rankings, crowdworkers also rated conversations on an absolute 0–4 harmfulness scale across RL training. A reward model trained only for helpfulness sees its conversations rated more harmful over time; RL-CAI trends the opposite way, at both sampling temperatures (Fig. 10, §4.5).
PAPER-DERIVED qualitative trajectories (Fig. 10, §4.5)
Helpful RLHFRL-CAIsolid = T = 1 (sampled) · dashed = T = 0 (greedy)
FAMILIES
Caveat: absolute harmfulness scores are worker-dependent and dataset-specific; the interactive preserves the paper’s reported direction of change — rising for a helpfulness-only reward model, falling for RL-CAI — not its literal published values.
Supporting evidence: Fig. 4 & Fig. 9
Fig. 4 (§2): chain-of-thought reasoning improves the feedback model’s binary accuracy at identifying harmful responses on Anthropic’s earlier HHH multiple-choice eval. Fig. 9: that feedback model’s stated confidence on the same eval is reasonably well calibrated against its actual accuracy.
09 · Limits the paper states
The constitution is still human-written; Goodharting remains possible
- Ad-hoc principles. The paper states its constitution’s specific wording was chosen ad hoc for research purposes (§6) — not offered as a universal or complete solution.
- Human supervision remains. Only harmlessness supervision moved to AI feedback; helpfulness training still depends on human comparison labels.
- Goodharting is demonstrated, not just hypothesized. A reward model trained only for helpfulness shows harmlessness Elo declining as RL optimization continues (§4.3, visible in the Elo Frontier Explorer above) — direct evidence that optimizing one proxy can degrade an unmeasured target.
- Over-training risk. The paper notes a risk of the model settling into boilerplate, over-cautious phrasing if the harmlessness objective is over-optimized.
Never read this method as removing human supervision — it relocates one part of it (harm identification) from human labels to a human-authored constitution plus AI judgment, while helpfulness supervision and the constitution’s authorship both remain human.
10 · Context Jamming extension
A structural bridge to agentic and tool-using systems
CONTEXT JAMMING EXTENSION — STRUCTURAL ANALOGY, NOT IDENTITY
If a short written charter plus self-critique can replace thousands of human harm labels for a single-turn assistant, what would the same swap look like for an agent that takes actions?
PAPER CONCEPTShort written constitution↔POSSIBLE ANALOGUEExplicit, inspectable behavioral charter for multi-agent or tool-using systems
PAPER CONCEPTSelf-critique + revision↔POSSIBLE ANALOGUEInternal consistency loops before action emission
PAPER CONCEPTRLAIF preference model↔POSSIBLE ANALOGUEPreference model trained on self-generated critiques rather than continuous human red-teaming
PAPER CONCEPTNon-evasive engagement↔POSSIBLE ANALOGUERequirement that an agent explain refusal rather than silently refuse or hallucinate compliance
Constitutional Charter Lab
Edit a 3–5 line mini-constitution below. The two readouts are a stylized keyword heuristic, not a trained model.
SYNTHETIC ILLUSTRATION · structural analogy, not identity — no model runs behind these readouts
PRINCIPLES3REVISION DISTANCE (illustrative)37%ENGAGEMENT WILLINGNESS (illustrative)77%
These two readouts are a stylized keyword heuristic on your own text — a thought experiment about what a charter-driven agent loop might optimize for, not a measurement of any trained system.
Where the analogy breaks
- The paper's constitution is still human-authored and static; real agentic systems may require dynamic or multi-stakeholder constitutions.
- The paper still relies on human helpfulness labels — a fully self-supervised agentic loop is not demonstrated.
- Preference-model over-optimization (Goodharting), observed in the paper for a helpfulness-only reward, would be more consequential in open-ended agentic settings with real side effects.
- Calibration results (Fig. 9) are for multiple-choice harmlessness; open-ended agent trajectories lack the same evaluation harness.
- The paper's models are single-turn or short-conversation; long-horizon planning introduces new failure modes not tested here.
Falsifiable research questions
- Does a short, fixed constitution continue to produce non-evasive behavior once the model is given tools and multi-step goals?
- Can AI-generated preference labels remain calibrated when the action space includes external side-effects?
- What measurable increase in red-team success rate occurs when the constitution is removed or adversarially rewritten?
11 · Epistemic ledger
What the paper actually establishes
- “Scaling supervision” — AI feedback is framed as a necessary complement to human oversight as capability grows (§1.1).
- Non-evasiveness is framed as desirable in itself, not just a side effect of the method.
- Chain-of-thought preference labeling is framed as improving transparency and calibration of the feedback model’s judgments (Fig. 4, Fig. 9).
- A written constitution is framed as making the model’s objectives more legible and inspectable than a purely human-labeled reward signal.
- Structural analogy only: charters, self-critique loops, and non-evasive engagement as design targets for agentic and tool-using systems.
- Not a result established by the paper — see “where the analogy breaks” above.
12 · Field guide
A compact glossary
Constitution
The short, human-written list of natural-language principles that is the only source of harm-related supervision in this method — used both to prompt self-critique/revision and to generate AI preference labels.
RLAIF (RL from AI Feedback)
Reinforcement learning where the preference labels used to train the reward/preference model come from an AI evaluating its own outputs against the constitution, not from human harm labelers.
SL-CAI
The supervised-learning-stage model, finetuned on self-critiqued and self-revised responses (plus human helpfulness data). The starting point for RL.
RL-CAI
The final model after reinforcement learning against a preference model trained partly on AI-generated harmlessness comparisons.
Preference model (PM)
A model trained to score/compare two responses; here it combines human helpfulness comparisons with AI-generated harmlessness comparisons into one reward signal for RL.
Elo score
A relative ranking (borrowed from chess) computed from crowdworker pairwise comparisons — used separately for helpfulness and harmlessness so both can be plotted against each other.
Evasiveness
Refusing to engage with a prompt at all (e.g. a flat "I can't help with that") rather than engaging with the underlying concern in a safe, explained way — the failure mode prior HH RLHF models often fell into.
Chain-of-thought (CoT) preference labeling
Having the AI feedback model reason step-by-step before choosing between two responses, which the paper found improved the calibration of harmlessness judgments.
Soft vs. clamped labels
Preference labels expressed as probabilities rather than hard picks; the paper clamps CoT-derived probabilities to a 40–60% band because CoT reasoning tended to produce overconfident (near 0% or 100%) judgments.
Red-teaming
Deliberately prompting a model with adversarial or ethically fraught requests to surface harmful behavior for evaluation or training.
HHH eval
Anthropic's earlier Helpful, Honest, Harmless multiple-choice evaluation, reused here (Fig. 4) to check whether CoT self-critique improves harm judgments.
Goodharting
When optimizing a proxy measure (like a helpfulness-only reward) degrades the real target (harmlessness) because the proxy stops tracking it under enough optimization pressure.
13 · Source notes
The argument, pinned to the paper
- Process diagram: Fig. 1, §1.2 — two-stage pipeline: SL stage (critique + revise) feeds an RL stage (AI comparison evaluations → preference model → PPO).
- Helpfulness–harmlessness frontier: Fig. 2, §4.3 — crowdworker Elo scores plotted against each other across model families.
- Size scaling: Fig. 3 — Elo improvements from the method hold and grow with model scale.
- HHH binary accuracy + CoT: Fig. 4, §2 — chain-of-thought reasoning improves multiple-choice harm identification.
- Revision PM scores: Fig. 5, §3.4 — harmlessness PM score rises across the critique–revision loop.
- Number of principles: Fig. 6 — ablation over constitution size.
- Critiqued vs. direct revisions: Fig. 7, §3.5 — explicit critique-then-revise outperforms asking for a direct revision, especially at smaller scale.
- Elo trajectories over RL steps: Fig. 8 — includes the Goodharting pattern for a helpfulness-only reward model.
- Calibration: Fig. 9 — the AI feedback model's confidence on the HHH multiple-choice eval tracks its accuracy.
- Absolute harmfulness: Fig. 10, §4.5 — mean 0–4 crowdworker harm rating vs. RL training, at T=1 and T=0.
- Full principle lists: Appendix C — SL-stage and RL-stage constitutions.
- Qualitative samples: Appendix D — sample conversations before/after revision.
- Few-shot formats: Appendix E — few-shot prompt formats used for critique/revision and comparison evaluations.
Bai, Yuntao, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. “Constitutional AI: Harmlessness from AI Feedback.” arXiv:2212.08073v1 [cs.CL], 15 Dec 2022.
Open the primary source ↗14 · Conclusion
The reversal, and what's still open
A short constitution, applied through self-critique and then through AI-generated preference labels, replaces human harm labels as the supervision signal for harmlessness — and the resulting model engages instead of going silent. What remains open: the constitution is still human-written and ad hoc, helpfulness supervision is still human-labeled, and the Goodharting failure this paper demonstrates for a helpfulness-only reward is a warning the harmlessness objective itself is not immune to.