Module 8 · suggested week 8
Distinguish genuine connections from beautiful nonsense.
What you will be able to do
Build a portable habit for locating the exact point where a scientific argument, or an LLM’s synthesis, outruns its evidence: hypotheses vs. evidence, confounding, selection, overfitting, extrapolation, preprints, benchmark contamination, and AI fabrication.
A hypothesis is a claim that could be true. Evidence is an observation that is more likely if the hypothesis is true than if it is false. That ratio is the whole game. In Bayesian terms (Module 4) the strength of evidence is the likelihood ratio . Evidence that would have appeared anyway is not evidence.
Every claim in this curriculum carries one of six status badges: established, empirical approximation, model, theory, analogy, speculation. The toolkit below is how you decide which badge a claim you meet in the wild deserves.
A test that is 90% sensitive sounds conclusive until the base rate is low. Move the prior and watch how much a positive result should actually move you. The same arithmetic applies to "a striking cross-field connection": if most striking connections are coincidences, a striking one is still probably a coincidence.
The most common failure in interdisciplinary writing is promoting shared mathematics (Level 2) to shared mechanism (Level 3). The tell is a verb change: "can be described by the same equation" quietly becomes "is governed by the same law" becomes "is the same phenomenon".
A mechanism claim needs three things that notation cannot supply: an identified causal process, a demonstration that the process operates in both systems, and a prediction that would differ if it did not.
The same differential equation describes a mass on a spring, an LC electrical circuit and (approximately) a pendulum at small angles. Engineers exploit this daily. No one concludes that springs are made of capacitors.
Correlation vs. causation. Two variables move together. Possible stories: A causes B, B causes A, a third variable C causes both (confounding), or the correlation was manufactured by how the data were gathered (selection bias).
Confounding in AI. "Models with more parameters reason better" — but larger models were also trained on more and better data, with more compute and newer techniques. Scaling papers handle this by varying one factor at a time in controlled sweeps; product comparisons across companies rarely can.
Selection bias in AI. "Every model that crossed X parameters showed ability Y" may only include the models that someone chose to evaluate on Y, or reports that someone chose to publish.
A p-value answers a narrow question: if there were no real effect, how surprising would data this extreme be? "Statistically significant" (often ) means "not very surprising under no-effect, by a conventional threshold". It does not mean the effect is large, important or probably real.
The effect size says how big the difference is. A tiny effect can be highly significant with enough data; a large effect can miss significance with too little.
Replication is the test that matters. Ioannidis (2005) argued, with a simple probability model, that when studies are small, effects are small, many hypotheses are tested and flexibility in analysis is high, most claimed findings can be false. Later large-scale replication projects in psychology and other fields found substantial failure rates, broadly consistent with his concern.
Ioannidis’s argument is a model: given assumptions about prior plausibility, power and bias, it computes the probability that a "significant" finding is true. Its conclusion depends on those inputs. Its lasting contribution is the habit of asking what they are.
Source: Ioannidis 2005, PLoS Medicine
In ML, "replication" often means rerunning with different random seeds, data orders or hyperparameters. Many benchmark gains are within seed-to-seed variance. A result reported from one seed, without error bars, is weak evidence. Look for multiple runs, confidence intervals, and ablations that remove the claimed ingredient.
Overfitting: a model (statistical or human) that fits the noise in its data, not just the signal. It looks perfect on what it was built from and fails on anything new. The tell is many free parameters or many choices relative to the amount of data.
Extrapolation: using a relationship outside the range where it was measured. Scaling laws are fitted over a range; extending the line ten times further is a prediction, not a measurement. Kaplan et al. explicitly noted their trends must eventually level off, since loss cannot fall below the entropy of natural language. Chinchilla (Hoffmann et al. 2022) then revised how compute should be split between model size and data.
Fitted scaling laws are empirical approximations with a valid range. Extrapolations beyond the measured range are hypotheses. They have often held approximately for loss, and they say nothing direct about specific downstream capabilities.
Fit by eye over a short range, then extend the axis. A power law and a power law plus a floor are nearly indistinguishable early on and diverge sharply later. That is the extrapolation trap in one picture.
Wei et al. (2022) catalogued "emergent abilities": tasks where performance was near chance for smaller models and jumped sharply above a scale threshold. Schaeffer, Miranda and Koyejo (2023) offered an alternative: many jumps come from the metric. If per-token accuracy improves smoothly but the task is scored by exact match over many tokens, the score is roughly per-token accuracy raised to the number of tokens, which stays near zero and then rises abruptly.
If each of tokens must be right and the model gets each right with probability (assuming independence), the whole answer is right with probability about . Raising smoothly from 0.8 to 0.95 takes a 10-token answer from about 11% to about 60%.
const L = 10;
for (const p of [0.5, 0.7, 0.8, 0.9, 0.95, 0.99]) {
console.log(p, Math.pow(p, L).toFixed(3));
}Does not say: Assumes independent token errors, a simplification. The paper tests the prediction empirically rather than relying on the toy model alone.
Same underlying improvement, two metrics. With the continuous metric, progress is smooth; with the all-or-nothing metric, an apparent "phase transition" appears. Illustrative curves generated from the formula above, not data from either paper.
Where the debate stands: Schaeffer et al. show that many reported emergent abilities become smooth under continuous metrics. That does not prove no capability ever appears abruptly, and it does not make discontinuous metrics illegitimate when the real-world task is all-or-nothing. This is ordinary scientific disagreement: both sides make testable claims about the same data.
Read it properly in the Mirage paper guide.
Preprints. arXiv is a distribution server, not a journal. Moderators check scope and basic form; no one checks whether results are right. Many landmark ML papers stay preprints forever, so "it’s on arXiv" is neutral: neither a badge of quality nor a red flag. Judge by methods, evidence and later replication. Conference acceptance (NeurIPS, ICML, ICLR) adds a light peer review, which is useful but noisy.
Benchmarks. A benchmark is a fixed test set that stands in for a capability. Two failure modes:
Evaluation design checks: held-out or freshly written test items; reported variance across runs; ablations; comparison to strong baselines; disclosed prompting and sampling settings.
Language models are trained to produce text that is likely given their training data. Cross-disciplinary syntheses are exactly where this goes wrong, because the training data contain many real connections (so the style is well learned) but few examples of the specific connection you asked about (so the content is interpolated).
Tells to watch for:
Language models can generate fluent, specific-sounding citations and technical claims that are false. Every citation and every "studies show" from a model must be checked against the primary source before use.
How to tell them apart
| Feature | Scientific disagreement | Pseudoscience / crankery | |
|---|---|---|---|
| Predictions | Both sides state what would change their mind | Claims adjust after the fact to fit any outcome | |
| Engagement | Responds to the strongest counterargument | Ignores or dismisses critics as closed-minded | |
| Scope | Claims are bounded: this setting, this range | One idea explains everything across all fields | |
| Mathematics | Equations derived or fitted, with assumptions stated | Equations used as decoration, with symbols and units that don’t check | |
| Track record | Builds on and is built on by others | Isolated; cites mainly itself | |
| Example | Emergent abilities: Wei et al. vs. Schaeffer et al. | A "unified theory" linking consciousness, quantum fields and LLMs with no testable claim |
A model that memorises its training set and an analyst who keeps adding epicycles to fit every anecdote fail the same way: flexibility outruns data.
Why this level: The bias–variance trade-off is a formal framework applicable to any estimator, human or machine. Applying it to human reasoning is informal.
A nonlinear readout of a smooth process can look like a sharp transition. This is relevant to physics-flavoured claims about sudden AI capabilities.
Why this level: Schaeffer et al. predicted which reported jumps would vanish under continuous metrics and tested that prediction. Whether every claimed jump is explained this way remains open.
Small samples, flexible analysis and publication incentives produce false positives in both fields.
Why this level: The statistical mechanism (low power + many tests + selective reporting) is genuinely the same; how strongly it applies varies by subfield.
Run any surprising claim (a headline, an abstract, an LLM synthesis) through these before you repeat it. The Skeptic’s Toolkit lab has worked examples where you find the breaking sentence yourself.
Bayes as restricting the space of possibilities, drawn as areas.
Watch for: The "Steve the librarian" example. It is a base-rate lesson in disguise.
Verified: YouTube oEmbed title+channel check, 2026-10-30
Bayes factors as "how much should this evidence move me". That is the most useful form for reading claims.
Watch for: Odds × Bayes factor. Use it on any "new study shows" headline.
Verified: YouTube oEmbed title+channel check, 2026-10-30
Gap repair for distributions, sampling, significance and study design.
Watch for: The study-design unit. Observational studies vs experiments is the correlation/causation line.
Verified: HTTP 200 + page content checked, 2026-10-30
Visual, interactive probability and statistics, from chance events to Bayesian inference.
Watch for: The chapter on Bayesian inference, as a second view on the Bayes interactive.
Verified: HTTP 200 + page content checked, 2026-10-30
The editorial benchmark for this curriculum: serious science, precise about what is shown versus hoped.
Watch for: Read with the Skeptic’s Toolkit open. Label each claim with the evidence badges used here.
Verified: HTTP 200 + page content checked, 2026-10-30
The most careful free treatment of power laws. It includes how to estimate an exponent and why straight lines on log-log plots are easy to over-read.
Watch for: The sections on fitting power laws. They are the antidote to "it is a power law, therefore it is universal".
Verified: HTTP 200 + page content checked, 2026-10-30
Why Most Published Research Findings Are False
Open access. A probability model of false discovery. Limitation: conclusions depend on assumed inputs; the title is stronger than the argument’s conditional form.
Emergent Abilities of Large Language Models
The catalogue of claimed emergent abilities. Limitation: many results use discontinuous metrics, as the next paper argues.
Are Emergent Abilities of Large Language Models a Mirage?
Metric choice can manufacture emergence. Limitation: shows many jumps are metric artefacts, not that all are.
Training Compute-Optimal Large Language Models
Revised Kaplan-era compute allocation: scale data with parameters. A model case of a scaling law being corrected by a better experiment.
Deep Double Descent: Where Bigger Models and More Data Hurt
Test error can rise, then fall, as size grows, challenging naive overfitting intuitions. Limitation: most visible with label noise and specific setups.
There are no right answers to check against. These questions are for thinking on the page, and your answers go to your notebook.
Find the last AI research claim you shared or nearly shared. Which evidence badge would you give it now?
Ask an LLM to connect two unrelated papers you know well. Mark the first sentence where it outruns either paper.
In your GTM work, where could a metric choice turn a smooth trend into an apparent breakthrough, or hide one?
What would convince you that a cross-field connection you like is wrong?
Bayesian updating
Bayes’ rule, , counted out. Among everyone who tests positive (filled squares), what fraction actually has ? When the prior is small, false positives from the large “not-H” group can swamp the true ones. Read “H” as “this research hypothesis is true” and the same arithmetic explains why surprising single studies often fail to replicate.
Power-law explorer
Illustrative constants, not a fit to real models. On log–log axes the reducible part, , is a straight line with slope −α. The full curve bends toward the floor , the loss that no amount of scale removes in this model. On linear axes nearly all of the change happens at the far left, which is why scaling papers plot on log axes.
The emergence mirage · same model, two metrics
Schaeffer, Miranda & Koyejo (2023, arXiv:2304.15004) argue that many “emergent abilities” come from the metric, not the model. If per-token accuracy p improves smoothly with scale, an all-or-nothing metric like exact match on an L-token answer behaves like pL, which stays near zero and then shoots up, looking like a sudden jump. Increase L and the “jump” gets sharper and later, with no change to the underlying model. The curve here is illustrative, not real data. It shows that metric choice must be ruled out before claiming emergence. It does not prove that every reported emergent ability is a mirage, which remains debated.