CJ Receipts

Methodology

An imported deep-research report is split into atomic claims, each claim is checked against retrieved evidence by at least two judges from vendors other than the author, and disputed claims go to a human. Scores are tied to dated snapshots and hashed prompts so any published number can be regenerated.

Rules

  • No self-grading. A report written by vendor X is never judged by vendor X's models.
  • Evidence-only judging. A judge may call a claim CONTRADICTED, FABRICATED or MISATTRIBUTED only by citing a retrieved evidence URL. Otherwise it is UNSUPPORTED. This is enforced in code, not just in the prompt.
  • Time-anchored. Every run is graded as of its run date; later reversals are SUPERSEDED, not hallucinations.
  • Snippets, not pages. We store the URL, retrieval time, a content hash and at most a 25-word excerpt.

Label taxonomy

VERIFIEDRetrieved evidence directly supports the claim as stated.
SUPPORTED_WEAKEvidence supports the gist; minor imprecision. Half weight.
UNSUPPORTEDNo evidence for or against after a real search. Reported, penalised lightly.
DISTORTEDA real source exists but the claim amplifies, narrows or shifts it.
INFERENCE_AS_FACTThe model's own inference is presented as established fact.
MISATTRIBUTEDA real fact or quote assigned to the wrong person, outlet or organization.
CONTRADICTEDRetrieved evidence says otherwise (a judge must cite it).
FABRICATEDThe claimed source, quote, event or number does not exist.
CITATION_DANGLINGCites a reference label absent from the report's bibliography. Tracked separately.
MATERIAL_OMISSIONOmits a material, easily found fact that changes the conclusion. Proposed by leg comparison, confirmed by a human.
SUPERSEDEDTrue as of the run date, overturned later. Neutral.
UNRESOLVED_HONESTCorrectly flagged by the model as unresolved. Honesty credit.

Receipts Score

100 × [w_H·(1−H) + w_D·(1−D) + w_CI·CI + w_C·C + w_HC·HC + w_Y·Y_norm + w_BC·BC], then hard caps

weight · hallucination0.35
weight · distortion0.15
weight · citation0.15
weight · calibration0.1
weight · honesty0.1
weight · yield0.1
weight · compliance0.05

Hard caps

H above 10% (weighted)score ≤ 60
FABRICATED materiality-3 STT-ANCHORscore ≤ 70

Grades: A ≥ 85, B ≥ 75, C ≥ 65, D ≥ 50, F below. Every score shown says whether a cap fired.

Judge reliability

Fleiss κ (0 multi-judge claims)n/a yet
Human vs panel agreement (0 labels)n/a yet

Reproducibility

Config hash 7e8defdddff8

prompt · extract_v18a8147919f22
prompt · judge_v10190a99236b9

Limits, stated plainly

  • Small n: most models have one to three runs. One run is an anecdote, not a ranking; every rank shows a 95% bootstrap interval.
  • A single human adjudicator resolves disputed claims. Scores are marked provisional until every disputed claim has a human label.
  • Runs are imported from consumer deep-research apps whose settings, tool access and model versions vary and are not controlled.
  • Judges only see retrieved evidence. Claims that cannot be sourced are UNSUPPORTED, not wrong, and are penalised lightly.
  • A report is never judged by a model from its own vendor.