FEATURES
AS
REWARDS

01 · Scalable supervision used to mean one thing: a bigger, slower judge.

The Supervision Bottleneck

Understand why external LLM-judge supervision has a built-in ceiling on cost, speed, and calibration for open-ended tasks.

Open-ended generation — long-form answers, agent trajectories, anything without a single verifiable scalar reward — has mostly been supervised by pointing another large model at the output and asking it to judge quality. That judge call is slow, expensive at scale, and only as well-calibrated as the judge model itself; it also produces one scalar per rollout, which is a thin signal for a policy trying to learn which specific span went wrong. The paper’s starting premise is that this bottleneck is not fundamental — it is a consequence of where the signal is read from.

02 · The model already knows more than it says.

Features Already Track What Matters

See the interpretability anomaly that makes this reframing possible: probes on hidden states already recover the right structure, even when the model’s own text does not.

On Longfact++-style long-form factuality data with the Gemma-3-12B-IT policy, the paper trains probes that read hidden states directly and classify whether a generated claim is supported. Four such probes reach AUC 0.88–0.94 with good calibration — accurate and well-behaved as decision instruments, not just descriptively correlated with ground truth (Sec. 3.1 / App. E).

03 · Stop asking a second model to judge. Start reading the first model’s own beliefs.

The Reframing: Features as Rewards

04 · Localize first, then classify — in that order.

The RLFR Pipeline at a Glance

Grasp the two-stage decomposition that makes probing reliable enough to reward on: find the span first, then judge it.

The pipeline the paper calls RLFR (Reinforcement Learning from Feature Rewards) runs a localization probe over the policy's hidden states to find candidate spans, then a classification probe to decide whether that span is supported. The authors argue this decomposition is critical for reliable monitoring — a single probe asked to do both jobs at once is a harder, noisier target than two probes each doing one job (Sec. 3.1). The resulting per-span signal then feeds retraction and correction reward probes (next section), and Longfact++ on Gemma-3-12B-IT is the substrate throughout.

05 · Where in the sentence, and is it true — answered by two different probes.

Monitoring: Localization & Classification Probes

Explore the probe operating point the paper actually reports, and see how precision and recall trade off around it.

Appendix E reports one concrete operating point at decision threshold 0.7: precision 0.85 / recall 0.56 for inline monitoring, and precision 0.88 / recall 0.61 for not-inline monitoring (Fig. 3). Move the threshold in the instrument below to see how that single measured point sits inside a broader, illustrative precision/recall trade-off, and inspect a binned calibration curve alongside it.

06 · Fixing a claim and retracting it are different skills; the paper rewards them separately.

Rewarding Interventions: Retraction & Correction Probes

Simulate the moment an unsupported claim gets caught, and see how the reward rubric scores what the policy does next.

Once a span is flagged unsupported, the policy can maintain it, retract it, or correct it. The paper trains separate retraction and correction reward probes rather than one combined signal, on the argument that clean removal and accurate replacement are distinct behaviors that a single probe would conflate (Sec. 3.2, App. A.3–A.4). The simulator below maps a chosen action, on a Longfact++-style claim, to the paper's reward-rubric categories.

07 · Multiply three signals; constrain the mix toward roughly 60:40.

How the Overall Reward Is Computed

Manipulate the three multiplicative reward components and watch a Lagrangian constraint pull the corrections-to-retractions mix toward its target.

Section 3.4 composes the overall reward as legibility × substantiveness × feature-reward, then applies a Lagrangian constraint so the policy doesn't degenerate into pure retraction (easy, low-value) or pure correction-attempts (risky, sometimes wrong) — steering the mix toward roughly 60% corrections to 40% retractions.

08 · Most of the gain lands well before training ends.

Training Dynamics: How the Policy Improves

Scrub through RL training and watch policy-level reduction and the fixed-plus-retracted rate both saturate around the same point.

Across RL training, the paper reports that the majority of the eventual gain arrives by roughly step 300 (Fig. 6). Both curves below saturate near that milestone; drag the slider to see the shape.

09 · At small N a judge keeps pace. At large N, probe rewards pull away.

Test-Time Scaling: Best-of-N with Probe Rewards

Compare probe-reward and LLM-judge best-of-N selection as N scales from 1 to 256.

Best-of-N sampling selects the best of N candidate completions by a reward signal. The paper reports that probe-reward selection opens up a growing advantage over LLM-judge selection as N increases, reaching roughly 15 percentage points higher success at high N (Fig. 7 / App. F) — while each probe call costs roughly 90x less than a Gemini 2.5 Pro judge call for an equivalent rewarded intervention, which is what makes sampling at large N practical at all.

10 · 58% fewer hallucinations, and the paper shows its work on where that comes from.

End-to-End Impact: 58% Reduction, Broken Down

Toggle each component of the reduction on and off to see how policy improvement, in-context effects, and direct intervention quality each contribute.

At best-of-32 sampling, RLFR reduces hallucinated content on Longfact++ by 58% overall relative to the Gemma-3-12B-IT base policy. The paper decomposes that reduction into three components: 10 percentage points from policy improvement, 35 from the in-context effect of inline interventions, and 13 from direct intervention quality (Fig. 4).

11 · Fewer hallucinations, not fewer claims.

What the Policy Changes — and What It Does Not

Check the one number that rules out the easy explanation: the policy isn't just saying less.

A skeptical read of any hallucination-reduction result asks whether the model simply became more terse — fewer claims, trivially fewer wrong claims. Section 4.3.2 heads this off directly: average entity count per completion is essentially unchanged, about 63.6 for the base model versus about 63.4 for the RLFR policy, reported as statistically identical. The policy also preserves standard benchmark performance overall (Table 1), and KL divergence from the base model runs 10–20% higher specifically on tokens classified as not-supported — the policy is shifting behavior where it matters, not uniformly.

12 · Successful fixes look at the entity. Failed fixes look at the comma.

Where the Probes Actually Look

Compare stylized attention patterns between successful and failed fixes to see the qualitative signature the paper describes.

Figure 5 shows that when an intervention succeeds, probe attention concentrates on entity and relation tokens — the substantive content of the claim. When an intervention fails, more attention mass falls on punctuation and filler tokens, a pattern consistent with the probe (and the policy) losing track of what it is actually correcting.

15 · What was demonstrated, what was proposed, and what remains open.

Epistemic Ledger and Glossary

Internalize the three-layer separation and key terminology with paper-anchored definitions.

Glossary