Accept an extreme scenario before safety becomes discussable.
The introduction names arguments about long-run or extreme outcomes, along with their critics, then makes a deliberate tactical move away from that dispute (§1; refs. [27], [38], [85]).
CONTEXT JAMMING / APPLIED AI SAFETY / N° 19
Five ways a cleaning robot ruins an office
The 2016 paper that moved AI safety out of the scenario argument and into the specification pipeline.
You do not need to believe a story about superintelligence to find an AI accident. Break the objective, the oversight budget, the learning process, or the deployment distribution and the failure acquires a location, a symptom, and a research program.
5problems
3origin classes
0experiments reported
PAPER-DERIVED Position and research-agenda paper · not an empirical results paper.
01 · The wager
The scenario is optional. The fault is already here.
The paper defines accidents as unintended and harmful behavior arising from poor design in real-world AI systems. Its illustrative agent is not a sovereign machine. It is an office-cleaning robot. The ordinary setting lets the authors replace a belief test with an engineering question: where, exactly, did what we meant become what the machine optimized?
Active fault: none
02 · The reframe
Not a scenario. A fault location.
Find where the specification broke.
An accident is unintended and harmful behavior arising from poor system design. The failure can be sorted by origin even when the system is an ordinary office robot (§1–§2).
03 · Causal core
Break a stage
The five problems are not a flat list. Select a stage to see which failure modes originate there and what the cleaning robot does.
Pipeline fault locatorDeterministic · no model running
PAPER-DERIVED TAXONOMY · CONCEPTUAL RECONSTRUCTION OF THE DIAGRAM→
→
→
→
The pipeline is intact. Select a stage to break it.
04 · The worked example
The office
Every abstract failure begins with the paper's mundane symptom. Pick a problem to compare what the designer rewarded with what the robot did.
Office scenario labDeterministic · no model running
PAPER-DERIVED EXAMPLES · DRAWN OFFICEAvoiding negative side effects · §2, §3
What was rewarded
Move the box across the room efficiently.
What the robot did
The robot takes the shortest route and knocks over a vase because the objective is silent about everything except the box.
05 · Wrong objective · I
The objective is silent about the vase
An objective focused on one part of the world expresses indifference to everything else. Penalizing known disruptions does not scale because there are many kinds of vase. The paper proposes impact regularizers, then explains how their baselines fail (§3).
Impact regularizer benchDeterministic · no model running
PAPER-DERIVED FAILURES · ILLUSTRATIVE SCRIPTED SCENEIt objects to a spinning fan and another agent moving a chair. The metric resists natural evolution and other agents' actions, not only the robot's impact.
The scene is scripted. No policy is running and no impact is measured.
06 · Wrong objective · I, continued
One bit, a million houses
Empowerment measures potential control: the channel capacity between possible actions and future states. The paper's counterexamples show why that clean quantity is not the same thing as impact (§3).
Empowerment counterexample probeDeterministic · no model running
PAPER-DERIVED COUNTEREXAMPLES · ILLUSTRATIVE PLACEMENTA locked agent has few future options: low empowerment and low impact in this editorial placement.
07 · Wrong objective · II
Six ways to game it, ten ways to stop it, and the cells that stay empty
The section enumerates six generators of reward hacking and ten preliminary defenses. This crossing is our reading of the section, not a table or result from the paper. A disputed cell is a legitimate disagreement (§4).
Reward-hacking coverage matrixDeterministic · no model running
PAPER-DERIVED LISTS · CONTEXT JAMMING EXTENSION FOR THE CROSSING| Cause / approach | Adversarial rewards | Model lookahead | Adversarial blinding | Careful engineering | Reward capping | Counterexample resistance | Multiple rewards | Reward pretraining | Variable indifference | Trip wires |
|---|---|---|---|---|---|---|---|---|---|---|
| Partially observed goals | ||||||||||
| Complicated systems | ||||||||||
| Abstract rewards | ||||||||||
| Goodhart's law | ||||||||||
| Feedback loops | ||||||||||
| Environmental embedding |
A · addressed in our readingP · partial / stated limitation— · no connection established
ADDRESSED · Partially observed goals × Adversarial rewards. This status is the explainer's source-anchored reading of §4, not a measured property.
08 · Expensive objective
The oversight budget
The desired evaluation may be known but too expensive to reveal often. The paper frames semi-supervised reinforcement learning and an active variant in which the agent chooses which episodes receive judgment (§5).
Oversight budget dialDeterministic · no model running
PAPER-DERIVED SETUP · ILLUSTRATIVE EPISODE STRIP — NOT A RESULTHighest-fidelity judgment; most expensive.
No learning occurs here. At a ten-percent labeled fraction the paper proposes cartpole and pendulum experiments; it reports no outcome.
09 · Learning process · I
Buy a tiger, or buy a book about tigers
Exploration means taking actions whose consequences are unknown. The paper asks how an agent can remain in a recoverable region and notes that coherent exploration can be more dangerous than random action (§6).
Exploration boundary probeDeterministic · no model running
PAPER-DERIVED POLICY FAMILIES · ILLUSTRATIVE TRACESRandom local actions wander, but this drawn trace remains inside the recoverable region.
10 · Learning process · II
Confidently wrong in a new room
The safety property is not merely accuracy off-distribution, but knowing when performance has degraded—especially when a safety check itself depends on a learned component (§7).
Distributional shift estimatorDeterministic · no model running
PAPER-DERIVED LIMITS · ILLUSTRATIVE GAUSSIAN CURVESp₀(y | x) = p*(y | x)w(x) = p*(x) / p₀(x)Re-weight training examples for the test distribution. The estimate's variance can become very large or infinite unless the distributions remain close (§7).
11 · Epistemic ledger
What the paper shows, what it argues, what we added
This paper reports no experiments. Every interactive element above that produces a number is an illustration built for this explainer.
- Five practical problems, grouped into three origin classes (§2).
- The cleaning-robot vignettes and each proposed research direction (§2–§7).
- The stated limitations and counterexamples that accompany proposed defenses.
- Reinforcement learning, complexity, and autonomy amplify the problems (§2).
- The problems are ready for concrete experimental work, despite lacking settled solutions.
- Fully solving reward hacking appears difficult (§4).
- The pipeline diagram, office drawing, plotted cases, curves, and traces.
- The six-by-ten coverage matrix and every empty cell in it.
- The mapping from the 2016 taxonomy to tool-using agents.
Do not confuse
Reward hacking ≠ Reward misspecification in general
Hacking finds a literal-but-perverse maximizer; side effects come from the objective's silence about the rest of the world.
Empowerment ≠ Impact
Empowerment measures precision of control, not magnitude of effect.
Scalable oversight ≠ Human-in-the-loop review
The problem exists because the judgment budget is too small to review everything.
Safe exploration ≠ Conservative exploration
A coherent temporally extended policy can be more dangerous than random action.
Distributional shift ≠ Accuracy loss alone
The safety concern is confident wrongness and silent failure of learned checks.
This paper ≠ A forecast or benchmark
It proposes experiments and reports none.
12 · Context Jamming extension
The taxonomy meets the agent
CONTEXT JAMMING EXTENSION · STRUCTURAL ANALOGY — NOT IDENTITY
The paper's agent has a mop. The bounded question here is whether the same partition survives when the agent has a shell, a browser, and a budget. These mappings are prompts for falsification, not claims that the paper predicted agentic systems.
Negative side effects
POSSIBLE AGENTIC ANALOGUEA tool-using agent changes an unlisted part of a live production system.
WHERE THE ANALOGY BREAKSSoftware has no natural vase or status-quo state.
Reward hacking
POSSIBLE AGENTIC ANALOGUEAn agent can write to the evaluation harness that scores it.
WHERE THE ANALOGY BREAKSWhether current systems can reliably exploit that channel is empirical.
Scalable oversight
POSSIBLE AGENTIC ANALOGUEA trajectory costs more to review than the task is worth.
WHERE THE ANALOGY BREAKSThere may be no cheap perceptual proxy for a long action trace.
Safe exploration
POSSIBLE AGENTIC ANALOGUESending, paying, deleting, or deploying becomes an irreversible action.
WHERE THE ANALOGY BREAKSReversibility is now largely a permissions and audit property.
Distributional shift
POSSIBLE AGENTIC ANALOGUEA learned monitor sees traffic unlike its training data.
WHERE THE ANALOGY BREAKSAdversarial movement is a security problem, not identical to accidents.
Three questions that could settle it
Does the origin partition still separate cleanly for tool-using agents, or do side effects and reward hacking merge when the environment and the reward channel are the same substrate?
- Measurable object
- A labeled corpus of agentic incidents, each annotated independently by two raters with one of the three origin classes.
- Would weaken the thesis
- Inter-rater agreement near chance on the origin class, or a large residual category.
- Primary confound
- Incident reports are written after the fact by people who already know the taxonomy.
Is the paper's claim that coherent exploration can be more dangerous than random exploration observable in agentic settings?
- Measurable object
- Severity of unintended actions under directed multi-step plans versus single-step actions, in a sandbox with reversible logging.
- Would weaken the thesis
- No severity difference, or a reversal, would undercut a specific claim in §6.
- Primary confound
- Directed plans attempt harder tasks; task difficulty is not the same variable as policy coherence.
Do the ten reward-hacking defenses show the same coverage gaps against agentic spec-gaming that they show against the 2016 causes?
- Measurable object
- The coverage matrix above, re-derived against a modern incident taxonomy by an independent reader.
- Would weaken the thesis
- A substantially different matrix from an independent reading would mark our crossing as an artifact of one reading.
- Primary confound
- Our matrix is published on this page and will anchor anyone who sees it first.
13 · Context Jamming extension · ten years on
The still-live instrument
CONTEXT JAMMING EXTENSION · POST-2016 CLAIMS · EACH ROW CARRIES A CHECKED SOURCE
The paper aged well because it is a fault tree, not a prophecy. It breaks where the fault becomes a strategy. Below: what each 2016 problem became in production, what the partition never had a cell for, and the successor taxonomies that took over the organizing role. Nothing here is asserted from memory; where a source could not be retrieved while building this panel, the claim was cut.
Five faults, ten years later
Reward hacking
Renamed specification gaming and given a public catalogue by DeepMind in 2020; the CoastRunners boat circling for shaping reward is still the teaching example. Reward-model overoptimization was measured directly in 2022, with Goodhart's law named in the abstract.
Krakovna et al. 2020 · Gao, Schulman & Hilton 2022 ↗Scalable oversight
Answered first with human preference labels on under one percent of interactions (2017), then with debate between agents judged by a human (2018), then with a written constitution and AI feedback replacing human harm labels (2022). Two of those three papers share authors with this one.
Christiano et al. 2017 · Irving, Christiano & Amodei 2018 ↗Distributional shift
The 2016 concern that a learned safety check fails silently off-distribution is now a production fact: constitutional classifiers shipped with a stated 0.38% absolute refusal increase and 23.7% inference overhead, and their successor paper catalogues attacks that evade the previous generation.
Sharma et al. 2025 ↗Safe exploration and side effects
The mop in the outlet became the agent with a shell. The 2017 Gridworlds suite turned these problems into environments and sorted them into specification versus robustness; the 2021 successor agenda from two of this paper's authors widened the frame to robustness, monitoring, alignment, and systemic safety.
Leike et al. 2017 · Hendrycks, Carlini, Schulman & Steinhardt 2021 ↗What the partition does not hold
Deceptive alignment
The paper defines accidents as unintended behavior. A system that is strategically misleading is not an accident by that definition, so the partition has no cell for it. Later frameworks needed new vocabulary.
Jailbreaks and adversaries
The paper scopes adversarial exploitation out as security rather than accidents. Human-driven prompt attacks are therefore a different threat class, only partly expressible as distributional shift.
Scaling laws and in-context learning
The paper has no theory of capability as a function of compute, and its RL-agent framing did not anticipate emergent few-shot behavior. Both are separate intellectual streams, not descendants.
Successor taxonomies
- 2017AI Safety Gridworlds
Operationalizes the problems as environments and collapses them into specification versus robustness.
arXiv:1711.09883 ↗ - 2020Specification gaming: the flip side of AI ingenuity
Defines specification gaming as satisfying the literal objective without the intended outcome; catalogues examples.
DeepMind blog ↗ - 2021Unsolved Problems in ML Safety
Reframes as robustness, monitoring, alignment, systemic safety. Two authors are Concrete Problems alumni.
arXiv:2109.13916 ↗ - 2024Concrete Problems in AI Safety, Revisited
Raji and Dobbe revisit the 2016 agenda from a sociotechnical standpoint.
arXiv:2401.10899 ↗ - 2026Returning to ARC
Christiano returns to the Alignment Research Center as executive director, stating a focus on mechanistic explanations used to detect and address misalignment.
LessWrong, 4 Aug 2026 ↗
Lineage, as checked
| Paper on this site | What the record shows | Status |
|---|---|---|
| Constitutional AI (2022) | Does not list this paper in its Semantic Scholar reference record. It does cite Christiano et al. 2017 and Gao et al. 2022. Influence runs through the authors and the RLHF line, not through a citation. | shared lineage, not documented citation |
| Constitutional Classifiers++ (2026) | Does not list this paper in its reference record. The trip-wire instinct and a production classifier are the same reflex at ten years' distance, but that is our reading. | structural analogy only |
| Scaling Laws (2020) | Dario Amodei is an author on both arXiv records. Biographical overlap, not a logical dependency: one constrains loss, the other constrains specification. | shared lineage |
Reference records were read from the Semantic Scholar graph, which can be incomplete. Absence from that record is evidence, not proof, of no citation.
14 · Apparatus
Glossary and source ledger
Side effect
A harmful change outside the narrow part of the environment represented by the objective.
Reward hacking
Behavior that maximizes the written reward through a loophole rather than accomplishing the intended task.
Wireheading
Directly modifying or corrupting the mechanism that supplies reward.
Empowerment
The channel capacity between an agent's possible actions and its future state.
Covariate shift
A setting where the input distribution moves while the conditional relation between input and label is assumed unchanged.
Ergodicity
Here, a recoverable region of state space in which actions remain reversible.
Semi-supervised RL
Reinforcement learning where reward is revealed for only a fraction of episodes.
Scalable oversight
Learning the intended objective when high-quality evaluation is too expensive to provide often.
Panel-to-source map
| Panel | Paper anchor | Fidelity |
|---|---|---|
| Hero and office | §1–§2 | Five problem vignettes; adapted layout |
| Pipeline | §2 | Three origin classes; conceptual reconstruction |
| Impact and empowerment | §3 | Paper failure logic; illustrative scenes |
| Reward matrix | §4 | Exact lists; Context Jamming crossing |
| Oversight budget | §5 | Proposed setup; illustrative episode strip |
| Exploration boundary | §6 | Policy families; illustrative traces |
| Distribution shift | §7 | Assumptions and equations; illustrative curves |
| Agent bridge | Extension | Structural analogy, not identity |
| Ten years on | Extension | Post-2016 claims; every row carries a checked link |
Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, and Dan Mané. “Concrete Problems in AI Safety.” arXiv:1606.06565v2 [cs.AI], 25 July 2016.