- Co-occurrence of ICL-score jump, induction-head formation, loss bump, and PCA pivot, across 34 models of varying size and architecture.
- Architectural intervention (smeared keys) moves the phase change in the predicted direction and, for 1-layer models, causes it to occur at all.
- Direct ablation in small attention-only models: removing induction heads collapses almost all measured in-context learning.
- Mechanistic reverse-engineering of the QK/OV circuit and K-composition in small attention-only models.
CONTEXT JAMMING / MECHANISTIC INTERPRETABILITY
THE MECHANISM
OF PROMPTING
A circuit snaps into place — and in-context learning jumps with it
Anthropic’s 2022 paper argues that a specific, simple circuit — the induction head — forms abruptly during an early-training window, and that this single snap explains most of why prompting a language model works at all.
01 · Before the paper
The old mental model
Training loss falls smoothly. In-context learning — a model getting better at predicting late tokens than early ones — was assumed to improve gradually along with everything else, and to scale up smoothly with model size, the way most things in deep learning do.
02 · The anomaly
A bump appears in the loss curve
Zoom into training loss early on and, for any model with more than one attention layer, there is a small window where loss falls faster than the smooth trend around it — the only place in training where the loss curve is not convex. Averaged over billions of tokens, a visible bump means something widespread is happening to model behavior, not a microscopic quirk.
One-layer models never show this bump. That single fact — the paper’s first clue — points straight at a mechanism that requires composing two attention heads.
03 · A coordinate change
Defining an in-context learning score
To study this quantitatively, the paper adopts Kaplan et al.’s macro framing: instead of naming specific few-shot tasks, just measure how much better a model predicts a late token than an early one in the same context.
Evidence label: PAPER-DERIVED VALUES. The paper checks that its conclusions do not depend on the specific choice of 50 and 500 (Unexplained Curiosities section).
04 · The mechanism appears
What is an induction head?
On a repeated random sequence of tokens, an induction head does one thing: it finds the previous occurrence of the current token, looks at what came right after it, and increases the probability that the same thing follows now.
05 · Argument 1 · Macroscopic co-occurrence
Four things happen at once
Scrub through training. Watch the ICL score jump, the induction-head prefix-matching score jump, the loss bump appear, and a PCA-trajectory pivot happen — all inside the same narrow 2.5e9–5e9 token window, and only in models with more than one layer.
PAPER-DERIVED VALUES · relative shapes & timing source-accurate; y-values are illustrative reconstructions
ICL score
Prefix-matching score
Loss
INSIDE PHASE CHANGE (2.5e9–5e9)1-layer: ICL -0.10 nats, induction head none · 2-layer: ICL -0.23 nats, induction head forming · 3-layer: ICL -0.23 nats, induction head forming
Caveat: one-layer models never develop substantial ICL or induction heads (Argument 1). The exact token count of the phase change depends on dataset and model details.
Before this window, models sit at roughly 0.15 nats of in-context learning. After it, they settle near 0.4 nats — a jump the paper measures across 34 models of varying size, architecture, and dataset. Over 75% of a model’s final in-context learning ability forms inside this one window.
06 · Argument 2 · Macroscopic co-perturbation
Move the architecture, move the phase change
If induction heads are the mechanism, then making them easier to build should move the phase change in a precisely predictable way. The paper tests this directly with a “smeared-key” modification that lets a single attention layer approximate K-composition on its own.
ILLUSTRATIVE RECONSTRUCTION · exact curves illustrative; direction and existence of the shift are paper-derived (Argument 2)
DEPTH
Smeared-key modification
k_j = sigma(alpha) * k_j + (1 - sigma(alpha)) * k_(j-1)A trainable σ(α) ∈ [0,1] interpolates each head’s key vector between the current and previous token — making it easy for one attention layer to express K-composition.
PHASE CHANGE ≈ 2.50e9 tokensA vanilla 2-layer model undergoes the ordinary phase change once it learns to compose heads.
Caveat: this shows induction heads are a sufficient minimal mechanism for the large ICL jump. It does not prove they are the only mechanism in large models with MLPs.
07 · Argument 3 · Direct ablation
Remove the heads, watch ICL collapse
In small attention-only models, the paper can directly delete a head at test time and measure what breaks. Ablating induction heads removes almost all of the in-context learning gained during the phase change; other head types barely move the needle.
PAPER-DERIVED VALUES · direction/magnitude source-accurate for small attention-only models (Argument 3)
Δ ICL from current ablations: 0.000 natsNo ablations applied. Baseline in-context learning curve shown in gray.
Caveat: ablations measure marginal effects. In models with MLPs, attention/MLP interactions may mask or amplify a head’s true importance — the paper treats this evidence as strong for attention-only models and only suggestive where MLPs are present.
08 · Argument 4 · Specific examples of generality
Literal copying, translation, and pattern matching — the same heads
Induction heads are defined narrowly: they copy literal repeated sequences of random tokens. The same heads, on real and synthetic text, also translate word-by-word and complete abstract categorical patterns.
CONCEPTUAL RECONSTRUCTION · attention/logit patterns illustrative but directionally faithful to the paper’s visualizations; scores are paper-reported (Argument 4)
| Head | Layer depth | Copying score | Prefix-matching score |
|---|---|---|---|
| Literal copying head | 21 / 40 | 0.89 | 0.75 |
| Translation head | 7 / 40 | 0.20 | 0.85 |
| Pattern-matching head | 8 / 40 | 0.69 | 0.94 |
Mr and Mrs Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal, thank you very much. They were the last people you would expect to be involved in anything strange or mysterious, because they just did not hold with such nonsense. Mr Dursley was the director of a firm called Grunnings, which made drills. He was a big, beefy man with hardly any neck, although he did have a very large moustache. Mrs Dursley was thin and blonde and had nearly twice the usual amount of neck, which came in very useful as she spent so much of her time craning over garden fences, spying on the neighbours. The Dursleys had a small son called Dudley and in their opinion there was no finer boy anywhere. The Dursleys had everything they wanted, but they also had a secret, and their greatest fear was that somebody would discover it. Mr and Mrs Dursley, of number four, Privet Drive, were proud to say that they were perfectly normal.
Caveat: these heads satisfy the paper’s narrow empirical definition of induction heads on random sequences while also exhibiting more abstract behavior on natural and synthetic text — the comparison is not a metaphor.
09 · Argument 5 · Mechanistic plausibility
How the circuit actually works
For small attention-only models, the paper can reverse-engineer the induction-head circuit at the parameter level: a previous-token head shifts key information forward one position, and the induction head’s QK circuit reads that shifted key via K-composition, then copies via a positive-eigenvalue OV circuit.
ILLUSTRATIVE MODEL · reverse-engineered for small attention-only models; existence of the same mechanism in large models with MLPs is correlational, not proven (Argument 5)
Residual stream
x_j = embed(token_j) + positionsThe residual stream carries a running sum of everything written by earlier layers — a shared bus every head reads from and writes to.
Caveat: full reverse-engineering of induction heads in large models with MLPs remains beyond current mechanistic-interpretability techniques.
Independent replication
Redwood Research, 2022Substituting the previous-token head’s idealized attention pattern recovered 99% of the ablation loss difference; a simple induction-head approximation recovered 65%; adding fuzzy OV-circuit matching recovered another 10%.
10 · Argument 6 · Continuity
Where the evidence runs out
The case gets weaker as models get larger. For small attention-only models, evidence is strong and causal. For large models with MLPs, it is correlational: the same phase change, timing, and induction-head formation appear, but the paper cannot ablate a 13B-parameter model’s way to certainty.
The most plausible alternative: larger models have more heads, and other composition mechanisms may form during the same window. If several “composition heads” all snap into place together, non-induction mechanisms could account for more of the phase change above some size than induction heads do.
| Layers | Parameters | d_model | Heads/layer | Head dim |
|---|---|---|---|---|
| 4 | 13M | 512 | 8 | 64 |
| 6 | 42M | 768 | 12 | 64 |
| 10 | 200M | 1280 | 20 | 64 |
| 16 | 810M | 2048 | 32 | 64 |
| 24 | 2.7B | 3072 | 48 | 64 |
| 40 | 13B | 5120 | 40 | 128 |
Curiosity the paper flags but does not resolve: the ICL score stays at roughly the same ~0.4 nats regardless of model size, from a 2-layer model to the 13B-parameter one — as if the “amount” of in-context learning saturates at a fixed relative gap, even though larger models are better at predicting every token index.
11 · Context Jamming extension
Phase changes, emergent abilities, and safety
STRUCTURAL ANALOGY, NOT IDENTITY
If one specific, interpretable circuit can produce a large, persistent capability jump this abruptly, what does that imply about capabilities we haven’t found the circuit for yet?
PAPERAbrupt formation of a specific, interpretable circuit during training↔SAFETY ANALOGUESudden emergence of a complex behavior or misaligned optimization pattern in a frontier system
PAPERA ~2.5e9–5e9 token window, 1–2% of the way through training↔SAFETY ANALOGUEAn unknown, possibly much later or much earlier, point in training or fine-tuning
PAPERDirectly observed via ablation in small attention-only models↔SAFETY ANALOGUENot directly observed for any specific hypothesized failure mode
Phase-change stress test
Drag the training horizon. The illustrative capability strength follows the same sigmoid shape the paper measured for ICL score — nothing here is a trained model.
SYNTHETIC ILLUSTRATION · structural analogy, not identity — no safety-relevant training run underlies this control
Illustrative capability strength: 42%Emerging abruptly — the same shape as the paper's ICL-score sigmoid, mapped onto a hypothetical capability.
This is a thought experiment, not a simulation of any real model or training run. It borrows the paper’s sigmoid shape to make a point about abrupt emergence — nothing more.
Where the analogy breaks
- The paper studies a known, interpretable circuit; real emergent misbehavior may have an entirely different internal structure.
- Training dynamics here are next-token prediction; safety-relevant mesa-optimization would require different incentives and a different objective.
- The paper’s phase change is early and produces a stable capability; later emergent behaviors in larger systems may involve different, less-understood mechanisms.
- No inner-alignment or reward-hacking experiment is performed anywhere in this paper.
- Large models may have multiple redundant mechanisms for a given capability, unlike the single clean circuit studied here.
The paper’s own Discussion section raises this directly: it did not find evidence of mesa-optimization, but notes that phase changes make future problems “more challenging for researchers and society to prepare for,” precisely because behavior can change discontinuously.
Falsifiable research questions
- Can we detect the formation of induction-head-like circuits in frontier models before they exhibit new few-shot capabilities on held-out tasks?
- Does ablating or regularizing induction-head formation change the timing or nature of other emergent behaviors?
12 · Epistemic ledger
What the paper actually establishes
- Induction heads likely account for the majority of in-context learning even in large models with MLPs.
- The same heads generalize from literal copying to “fuzzy nearest-neighbor” pattern completion.
- No evidence of mesa-optimization was found — but the search was limited to this one mechanism.
- Structural analogy only: phase changes as a lens on emergent-capability safety risk.
- Not a result established by the paper — see “where the analogy breaks” above.
13 · Field guide
A compact glossary
In-context learning (ICL)
The tendency of a transformer to predict later tokens in a sequence better than earlier ones, using context alone, without changing weights.
ICL score
The paper's heuristic: loss at the 50th token in context minus loss at the 500th token, averaged over examples. Higher means more in-context improvement.
Induction head
An attention head that, on a repeated random sequence [A][B]...[A], attends back to the token after the earlier [A] and increases the logit for [B].
Prefix matching
The property that a head attends to the token that induction would predict comes next — i.e. the token following an earlier occurrence of the current token.
Copying
The property that a head's output increases the logit of the token it attends to.
Phase change
A narrow, abrupt window early in training (roughly 2.5e9–5e9 tokens in the models studied) where ICL score, induction-head formation, a loss-curve bump, and a PCA-trajectory pivot all occur together.
Previous-token head
An attention head that copies information about token j-1 into the residual stream at position j, feeding the induction head via K-composition.
K-composition
A mechanism where one head's query-key circuit reads from a subspace written by an earlier head's output — the way induction heads read "the preceding token" information.
QK circuit
The query-key portion of an attention head, which determines its attention pattern (where it looks).
OV circuit
The output-value portion of an attention head, which determines what it writes once it attends somewhere. Positive eigenvalues correspond to copying.
Smeared-key architecture
An architectural modification that interpolates each head's key vector with the previous token's key, making it easier for any depth of transformer to express induction heads.
Per-token loss analysis / PCA
A method that collects a vector of losses across many fixed tokens for each training snapshot, then applies PCA to visualize how model behavior evolves during training.
Attention-only model
A transformer with attention layers but no MLP layers — the setting where the paper has the strongest mechanistic and causal evidence.
Mesa-optimization
A hypothesized situation where a trained model develops its own internal optimization process. The paper looked for, and did not find, evidence of this.
Ablation (head knockout)
Removing a specific attention head at test time and measuring the resulting change in model behavior — the paper's main causal-evidence tool for small models.
14 · Source notes
The argument, pinned to the paper
- Abstract & thesis: Abstract, Introduction; six arguments listed and summarized in the “Summary of evidence” table.
- Key concepts, ICL score, induction-head definition: “Key Concepts” section (In-Context Learning, Induction Heads, Per-Token Loss Analysis).
- Argument 1 (phase change, co-occurrence): “Arguments that induction heads are the mechanism...” → Argument 1, including the loss/ICL/prefix-matching/PCA figures for 1-/2-/3-layer attention-only models.
- Argument 2 (smeared-key architecture): Argument 2 section and equation k_j = σ(α)k_j + (1−σ(α))k_(j−1); Model Details → Smeared Key Models.
- Argument 3 (ablation): Argument 3 section; ~250% ICL improvement figure (−0.15 → −0.4 nats).
- Argument 4 (generality — translation, pattern matching): Argument 4, Behaviors 1–3 (literal copying, translation head layer 7/40, pattern-matching head layer 8/40), example-head table.
- Argument 5 (mechanism, K-composition, QK/OV): Argument 5, “Summary of Reverse Engineering Induction Heads”; Redwood Research and Tom Lieberum replication comments.
- Argument 6 & Model Analysis Table: Argument 6; Model Analysis Table and Model Details (parameter counts, d_model, heads per layer).
- Safety discussion: Discussion → Safety Implications (phase changes, in-context learning, mesa-optimization).
- Unexplained curiosities: “Unexplained Curiosities” — constant ICL score across model sizes; derivative-order inversion at the phase change.
Olsson, Catherine, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Dawn Drain, Deep Ganguli, Zac Hatfield-Dodds, Danny Hernandez, Scott Johnston, Andy Jones, Jackson Kernion, Liane Lovitt, Kamal Ndousse, Dario Amodei, Tom Brown, Jack Clark, Jared Kaplan, Sam McCandlish, and Chris Olah. “In-context Learning and Induction Heads.” arXiv:2209.11895v1 [cs.LG], 2022.
Open the primary source ↗