Weight-Space Geometry
The Sixteen-Joint Machine
The useful content of a trained network doesn’t live in all its weights — it lives in about sixteen directions per layer, joints that every model, regardless of task or initialization, converges onto. The open question is no longer whether that subspace exists, but whether we can pretrain natively inside it instead of rediscovering it from scratch on every run.
01 · The finding in 60 seconds
An action figure has about sixteen points of articulation
A collectible action figure isn’t posable because of the plastic — it’s posable because of a small number of engineered joints molded into each limb: shoulders, elbows, hips, a handful of others, usually around sixteen. Every pose the figure can strike is just a setting on those same joints.
The paper’s discovery is that trained neural networks look the same way, except nobody designed the joints on purpose — they emerge from training itself. Across 1,100+ models — ViTs, LLaMA3, GPT-2, Mistral LoRAs, and ResNet-50s trained from scratch on entirely disjoint data — the majority of each layer’s useful weight variance concentrates into roughly sixteen shared directions. Different tasks, different random initializations, even different training datasets: same joints, different pose. Most strikingly, 500 randomly initialized ViTs, never trained at all, already sit on the same low-rank subspace.
How they found it
Truncated, zero-centered Higher-Order SVD (HOSVD) across each population of models, framed theoretically as a Hilbert-space second-moment operator. A two-level convergence theorem bounds the subspace error as it shrinks at O(1/√T) in the number of tasks analyzed, plus a per-task estimation term; how tightly that bound holds is governed by the eigengap γk between the k-th and (k+1)-th eigenvalues (Davis–Kahan). All of it ran on a single NVIDIA A5000 — extraction carries no training costs whatsoever.
02 · Where the joints are
Explained variance drops off a cliff around 16
Drag k to change how many of the top directions you keep. Almost everything worth having is captured early; almost everything past ~16 is what the paper calls the secondary subspace — directions that, when used on their own, perform drastically worse.
81.9%
of variance in the first 16 joints
18.1%
discarded — the leftover directions the paper shows perform drastically worse
03 · Pose the figure
Sixteen sliders, one shared skeleton
Each slider, c1 through c16, is a coefficient on one of the sixteen shared joints driving a live rank-16 reconstruction of an abstract weight-pattern grid. The presets show what the paper means by “different tasks are different poses”: same joints, entirely different fingerprint.
04 · The evidence
What holds up when you check the numbers
OOD projection & reconstruction
Unseen models, reconstructed inside the 16-dimensional subspace, checked against their full-parameter originals.
94.1% vs 94.4%
ViT · IID accuracy (16-dim vs full)
87.8% vs 91.3%
ViT · OOD accuracy (16-dim vs full)
5 unseen ViTs reconstructed in the subspace.
ResNet-50, five datasets, full → Universal R50
Coefficient-only training
Training just the ~16 coefficients per layer, with the shared joints frozen.
~10K
Trainable params, Universal ViT
vs 86M for the full model
84.01
GLUE avg, RoBERTa-base universal order-2
At 2× speedup vs LoRA's 83.67
19×
Mistral LoRA subspace memory efficiency
Universal ViT (~10K params) vs full ViT (86M params)
Model merging
Eight ViT tasks, merged into one model, scores normalized.
83.5
Ours, average across 8 tasks
68.0
Best baseline (KnOTS-TIES), average
88.1
Cars
82.3
DTD
65.9
EuroSAT
61.3
GTSRB
88.3
MNIST
98.1
RESISC45
98.5
SUN397
85.1
SVHN
Compression
Storing a population of models as one shared basis plus small per-model coefficients.
150 GB
500 ViTs, compressed
>100× cut
1.6 TB
LLaMA collection, compressed
>100× cut
19×
Mistral LoRA memory efficiency
19.83 vs 19.73
SDXL universal subspace CLIP score
Top 16 components, 30 denoising steps, vs individual style LoRAs
Ablation — diminishing returns
Mistral OOD Rouge-L, as the universal subspace is estimated from more source models.
73.7
Normal (full)
55.8
From 50 models
66.1
From 150
71.9
From 250
72.3
From 450
05 · Why it matters
Three implications, in order of how settled they are
Settled
Compression and reuse
Training ~10K coefficients instead of 86M parameters, storing 500 models in 150 GB instead of the original size, cutting Mistral LoRA memory by 19× — this is demonstrated, working today.
Open
The unbuilt leapfrog
Every universal subspace so far was extracted after training. The authors flag native subspace pretraining — building a model inside the 16 joints from the start — as future work, not something anyone has done.
Concerning
The monoculture risk
If every model collapses onto the same subspace, they may inherit the same biases and failure modes. The authors raise this themselves: is shared-basis convergence a diversity bottleneck?
06 · Honest limitations
Where to stay skeptical
- Shared-checkpoint critique. The ~500 ViTs pulled from HuggingFace were unfiltered, with no access to training data or provenance — they may share base checkpoints, which would make convergence look more universal than it is.
- Thin from-scratch evidence. The strongest from-scratch claim rests on 5 ResNet-50s trained on disjoint datasets and 500 randomly initialized (untrained) ViTs — a small sample for a universal claim.
- Interpretability is unproven. The paper itself describes interpreting what the 16 directions actually represent as “extremely cumbersome.” Nobody has confirmed what each joint does.
- Nobody trains inside the subspace yet. Model-independent, data-derived universal bases, and native pretraining inside them, remain future work — not a demonstrated result.
07 · Field guide
A compact glossary
Weight subspace
A low-dimensional set of directions in a layer's weight space that captures most of the variance across many trained models.
HOSVD (truncated, zero-centered)
Higher-Order Singular Value Decomposition — the tensor generalization of SVD the paper uses to find the shared low-rank directions across a population of models.
Coefficients (a pose)
The small set of numbers (here, ~16 per layer) that place one specific model at one specific point within the shared subspace.
Secondary subspace
The leftover, discarded directions beyond the top ~16 — shown to perform drastically worse, which is part of the evidence the chosen 16 actually matter.
Eigengap (γk)
The gap between the k-th and (k+1)-th eigenvalues of the second-moment operator; the paper's stability guarantee gets weaker as this gap shrinks.