280B parameters
300B training tokens
60.0%5-shot MMLU
Compute-optimal scaling
The model got smaller. The result got better.
Chinchilla changed the allocation, not the idea of scaling. A fixed compute budget could produce a better language model by spending much more of it on training tokens—and much less on parameter count.
The object to optimize was not model size. It was the allocation of compute between model size and data.
Gopher280B
→Chinchilla70B
Do not confuse
Chinchilla did not overturn the discovery that language-model loss scales predictably. It revised the prescription for how a fixed compute budget should be divided between parameters and training tokens.
Context Jamming / critical distinctionThe optimization target
The biggest model was never the question.
Training capacity arrives as a constraint: a fixed number of accelerators for a fixed amount of time. Hoffmann et al. asked how that budget should be divided before the expensive run begins.
The measured target is final pretraining loss. Lower loss here means better next-token prediction on the paper’s data distribution—not a general measure of intelligence.
Paper-derived
Nopt(C), Dopt(C) = argmin L(N,D)subject to FLOPs(N,D) = CGiven compute budget C, find the parameter count N and training-token count D that minimize final pretraining loss L.
Equation 1 · §1The primary instrument
One budget. Two ways to spend it.
Along a fixed-compute slice, increasing model size leaves fewer tokens available for training. Decreasing model size buys a longer training horizon. The fitted loss forms a valley between those two failure modes.
Fixed-compute allocatorFit-derived
Spend the same compute differently.
Move parameter count. Training tokens move in the opposite direction so that the approximation C ≈ 6ND remains fixed.
Budget C5.76 × 10^23FLOPs
Training tokens D1.37TC / 6N
Tokens / parameter19.6local ratio
Fitted loss1.9380.4% above fit min.
Paper-derived
C ≈ 6NDFLOPs are approximately proportional to parameters times tokens because each token requires work across the model during the forward and backward passes. The paper compares this approximation with its detailed accounting in Appendix F and finds little effect on the analysis.
§3.3 · Eq. 4 · Appendix F, Table A4Kaplan → Hoffmann
The laws survived. The exponents moved.
Both papers describe predictable power-law scaling. They disagree about allocation. Kaplan’s compute-efficient frontier sends a 10× compute increase mostly into model size; Hoffmann’s three approaches send roughly equal growth into parameters and data.
Nopt ∝ Cᵃ0.73parameter exponent a
Dopt ∝ Cᵇ0.27data exponent b
When compute grows 10×5.4× / 1.9×N growth / D growth
Additional compute goes predominantly toward model size.
Hoffmann et al., Table 2; Kaplan et al. (2020)
What survivedScaling laws
Loss remained smooth enough to fit and extrapolate within a measured regime.
What changedCompute allocation
Data had to scale much faster than the dominant 300B-token training pattern.
Approach 3
A loss surface with two shortages.
Paper-derived
L(N,D) = 1.69 + 406.4 / N⁰·³⁴ + 410.7 / D⁰·²⁸1.69 fitted irreducible termN term penalty for insufficient model capacityD term penalty for insufficient optimization/data
A giant model trained on too few tokens pays the D penalty. A tiny model trained for a very long time pays the N penalty. Under fixed compute, the minimum balances them.
Equation 10 · Appendix D.2Fit-derived
Values produced by the allocator evaluate this fitted equation. They are not additional training runs and are intentionally styled differently from the Chinchilla/Gopher measurements below.
The large-scale test
Then they trained the reversal.
The three fitted approaches placed the compute-optimal model for Gopher’s budget between roughly 40B and 70B parameters. DeepMind trained the upper end: Chinchilla, 70B parameters over 1.4T tokens.
70B parameters
1.4T training tokens
67.6%5-shot MMLU
51 / 57MMLU tasks won
2 / 57tied
4 / 57lost
Paper-derived · Tables 1, 6 & A7 · Figure 6
Causal caution
Powerful comparison. Imperfect ablation.
Chinchilla and Gopher share a reported training-compute budget, but model size and token count were not the only things that changed. The result validates the allocation prescription at large scale without isolating a single causal variable.
OptimizerAdam → AdamW
Appendix G reports AdamW performing better in smaller comparisons.
TokenizerModified SentencePiece
94.15% of tokens match; NFKC normalization was removed.
Data mixtureMassiveText resampled
The subset distribution changed to support the longer training horizon.
NumericsHigher-precision state
A float32 weight copy was retained in the distributed optimizer state.
Paper-derived · §4.1 · Appendix G
The strategic reversal
Bigger model and better use of compute are not synonyms.
Training outcome
At the same reported pretraining compute, the smaller, longer-trained model produced lower language-model loss and stronger measured downstream results.
Dataset priority
The authors argue that existing large LMs were substantially undertrained and that dataset scaling deserved more attention.
Capital allocation
Treat training compute as a portfolio: parameters buy capacity; tokens buy optimization. The visible asset is not automatically the best allocation.
Inference consequence
A 4× smaller parameter count also lowers memory and inference-compute requirements, extending the benefit beyond pretraining.
Conceptual companionOpen Kaplan 2020: When Scale Became a Forecast →
Scope and uncertainty
Twenty tokens per parameter is not a law of nature.
Objective boundary
The paper optimizes pretraining loss under a compute constraint—not capability, safety, economic value, latency, or every deployment objective.
Regime boundary
The scaling runs used autoregressive Transformers, mostly under one epoch, with measured model and data ranges. Extrapolation adds uncertainty.
Data boundary
More tokens are not automatically better tokens. The authors frame high-quality data as important and flag leakage, privacy, bias, and toxicity risks.
Frontier boundary
The paper observes concavity at high compute and says the power-law frontier may still overestimate optimal large-model size.
Evidence boundary
Only Chinchilla and Gopher form the comparable large-scale pair; there are no intermediate large-scale tests.
Ratio boundary
“20 tokens per parameter” summarizes a neighborhood of this fitted regime. Architecture, data quality, repeated epochs, and objectives can move the optimum.
Paper-derived + Context Jamming boundary synthesis · §5
Primary record
Read the claim at its source.
Primary source
Training Compute-Optimal Large Language Models
★ Equal contributions according to the paper.
- Organization
- DeepMind
- Record
- arXiv:2203.15556v1 · cs.CL
- Submitted
- 29 March 2022
- Type
- Empirical scaling-law / large-scale training study
- Venue
- arXiv preprint
Conceptual reconstruction
Charts redraw the paper’s argument using inline SVG and equations cited in each panel; they do not reproduce the paper’s figures or imply access to the underlying training-run dataset. Empirical benchmark values are separately labeled paper-derived.