CONTEXT JAMMING / EMPIRICAL LAWS OF AI
WHEN SCALE
BECAME A FORECAST
How model size, data, and compute collapsed onto a small set of power laws—and reorganized the logic of training language models.
Kaplan and colleagues trained language models across enormous ranges of scale and found something unexpectedly orderly. When no other resource became the bottleneck, test loss fell along smooth power laws in parameters, training data, and compute. Architecture still mattered—but within the tested Transformer regime, scale mattered far more.
01 · The measurement
A smooth surface under a messy practice
Before this paper, model performance looked like the residue of architecture choices, optimizer lore, training time, data volume, and engineering judgment. Kaplan et al. found that much of this apparent complexity collapsed when experiments were compared at scale.
The striking object was not a single winning model. It was a family of nearly straight lines on logarithmic axes: loss declining predictably as the limiting resource increased.
02 · Loss decoder
What the vertical axis actually measures
The paper measures autoregressive token-level cross-entropy in nats. Lower loss means the model assigns more probability to the observed next tokens. It is not a score for eloquence, reasoning, or intelligence.
A fall of 0.1 nat multiplies perplexity by e−0.1 ≈ 0.905. Small-looking changes compound.
03 · The training equation
Three scales—and two hidden gears
04 · Architecture inside the regime
Scale dominated shape—but did not erase architecture
Across the tested Transformer variants, performance depended mainly on non-embedding parameter count. Wide and deep shapes with similar N often landed near the same trend. This is a bounded empirical result—not permission to declare every architecture equivalent.
Within the tested Transformer family, loss tracked total non-embedding parameters more strongly than depth-versus-width shape.
05 · Power-law calculator
A shallow exponent is an expensive promise
On log-log axes a power law becomes a line. For compute, α ≈ 0.05 means that a hundredfold increase in optimal compute reduces the fitted loss by only about 21 percent. The slope is shallow; the predictability was the revelation.
06 · The N–D frontier
A larger model cannot eat data it never sees
The joint fit combines parameter-limited and data-limited terms. Increase N while holding D too small and the data term dominates; increase D around a tiny model and capacity dominates.
07 · Overfitting gauge
The data requirement grows sublinearly with model size
In the experiments, the onset of meaningful overfitting followed an approximate boundary D ≳ 5×10³N0.74. An 8× larger model required roughly 5× more data—not 8×—to stay near that boundary.
08 · The race
Bigger models learn from fewer examples
At equal test loss, larger models were more sample-efficient. But every token costs more to process in a larger model, so fewer examples does not automatically mean fewer FLOPs.
Larger models generally reached a given loss with fewer examples—but required more compute per example. Sample efficiency is not compute efficiency.
CONCEPTUAL RECONSTRUCTION09 · Learning-curve forecaster
Early trajectory could forecast the later run
The paper fit loss jointly as a function of model size N and update steps S. Once initial transients passed, the power-law learning curve offered a way to forecast how much improvement remained.
10 · Critical batch lab
Parallelism has a moving speed limit
The critical batch size rises as loss falls. Below it, larger batches can reduce the number of sequential updates; above it, extra batch mostly spends more computation for little wall-clock benefit.
The paper explicitly warns that the critical-batch fit is extrapolated outside parts of the observed loss range.
11 · Compute allocator
Spend the next FLOP on a larger model
Kaplan’s compute-efficient frontier assigned most marginal compute to model size: N ∝ C0.73, while processed data grew as D ∝ C⁰·²⁷. That prescription favored very large models stopped well before convergence.
12 · Convergence tradeoff
Training to completion could be the inefficient choice
For a fixed loss, Appendix B estimated a radically different operating point: a model 2.7× larger, trained for 7.7× fewer updates, using 65% less compute than the near-converged alternative.
At the same loss, Appendix B estimates that compute-efficient training uses a larger model, 7.7× fewer updates, and 65% less compute than training close to convergence.
13 · Distribution transfer
The slope may travel; the intercept may not
The paper found broadly similar scaling on several text distributions, with offsets between them. A familiar slope on a shifted distribution can still mean a worse absolute loss—and transfer beyond tested text domains was conjecture.
14 · Extrapolation collision
A straight line is not a warranty
Power laws invite projection. The farther a forecast travels beyond measured model sizes, data distributions, losses, and optimization procedures, the more hidden assumptions it crosses.
15 · Kaplan / Chinchilla
The allocation changed; the scaling worldview survived
In 2022, Hoffmann et al. revisited compute-optimal allocation with more extensive training and found model size and tokens should grow at roughly equal rates. Their 70B-parameter Chinchilla model used 1.4T tokens and outperformed the 280B-parameter Gopher at the same compute budget.
Later work found compute-optimal model size and training tokens should grow at roughly equal rates. That revises the allocation, not the original discovery that loss scaled smoothly.
HISTORICAL COMPARISON · DISTINCT FITSHow a fixed compute budget should be divided between parameters and data.
WHAT HELDLoss remained smooth enough across scale to support empirical forecasting.
16 · Why the paper mattered
Training became an allocation problem
Choose a model, tune it, train it, discover the outcome.
Choose a target loss and compute budget; estimate the efficient model and training horizon.
The paper did not make language models predictable in every important sense. It made one narrow but consequential quantity—test loss—forecastable enough to organize enormous capital and engineering decisions.
17 · Epistemic ledger
What is measured, modeled, and still unknown
Smooth loss trends across 7+ orders of magnitude in scale, inside the tested decoder-only Transformer regime.
Exponents and characteristic scales for separate N, D, C fits; joint surfaces; learning curves; and compute allocation.
Extension to other modalities, architectures, distributions, capability thresholds, and scales far beyond the experiments.
A solid theory explaining why these exponents arise. The paper calls the N and compute scaling especially mysterious.
18 · Glossary
The vocabulary of the scaling surface
- Cross-entropy loss
- The average surprise, in nats, assigned to the next token. Lower is better.
- Perplexity
- e raised to the loss: the effective branching factor implied by a model’s uncertainty.
- Power law
- A relationship of the form y ∝ x^a. It appears as a straight line on log-log axes.
- Scaling exponent
- The slope on a log-log plot. Small exponents mean improvement is steady but expensive.
- N
- The number of non-embedding model parameters in the Kaplan paper.
- D
- Dataset size, measured in training tokens.
- C
- Estimated non-embedding training compute; reported in PF-days in the paper.
- B
- Batch size in tokens processed per parameter update.
- S
- The number of parameter-update steps.
- Critical batch size
- The batch beyond which added parallelism yields diminishing returns for reaching a target loss.
- Compute-efficient frontier
- The allocation of model size and training duration predicted to minimize loss for a fixed compute budget.
- Early stopping
- Ending training before full convergence because a larger, less-converged model may use compute more efficiently.
- Sample efficiency
- How much data a model needs to reach a target loss; distinct from total compute efficiency.
- Bottleneck
- The resource currently limiting performance: parameters, data, or compute.
- Interpolation
- Prediction within or near measured support; safer than extrapolating far beyond it.
19 · Conclusion
Forecast the loss. Do not confuse it with the future.
Kaplan et al. showed that a complicated training system could exhibit remarkably simple aggregate behavior. The discovery helped turn scaling from an intuition into an operating model.
But the fitted variable was cross-entropy, the regime was specific, and the exponents were empirical. A scaling law can tell you how much more resource a measured trend demands. It cannot, by itself, tell you what capabilities will appear, whether the data remain adequate, or whether the same law survives a new architecture.
Scale became a forecast—not a theory of intelligence.
20 · Source notes
Fit families, kept separate
- One-dimensional fits: Appendix A, Tables 4–5: αN=.076, αD=.095, αC=.050.
- Joint N–D surface: §4, Eq. 1.5 and Table 2: αN=.076, αD=.103.
- N–S learning curve: §5 and Table 3: αN=.077, αS=.76.
- Critical batch: §5.1 and Appendix A: αB=.21, B*=2.1×10⁸ tokens.
- Compute frontier: §6 and Appendix A, Table 6: N∝C^.73; D∝C^.27; B∝C^.24; S∝C^.03.
- Early stopping: Appendix B.3–B.4.
- Caveats: Appendix C and §8.
Kaplan, Jared, et al. “Scaling Laws for Neural Language Models.” arXiv:2001.08361v1, January 23, 2020.