# Filtering Training Data to the Universal Weight Subspace

Boundary: one controlled mechanism check and one small-scale digits MLP. No ViT, LLM, or production-training result.

Framing: training that minimizes cross-entropy loss is training that minimizes a compression cost (Shannon 1948/1950); the gradient-alignment score below is a gradient-space view of that same statement.

Canonical URL: https://www.contextjamming.com/universal-weight-subspace-filtered-training
Author: Bret Kerr
Author profile: https://www.linkedin.com/in/bretkerr/
Source paper: https://arxiv.org/abs/2512.05117
Repository download: https://www.contextjamming.com/downloads/universal-weight-subspace/uws-filter-reproducible-repo.bundle (ZIP bytes)
Digits record: https://www.contextjamming.com/research/universal-weight-subspace/digits-results.json
Stability record: https://www.contextjamming.com/research/universal-weight-subspace/stability-results.json
Cost memo record: https://www.contextjamming.com/research/universal-weight-subspace/cost-memo.json
Companion essay (compression across domains): https://contextjamming.com/armchair-physics
Methodology colophon: this page documents its own build and verification pipeline.

## Question

If related trained models occupy a shared low-rank layer-wise weight subspace, can examples be scored by the fraction of their gradient energy inside that basis and then used for filtering, importance sampling, or hard update projection?

## Measured here

- Controlled planted-rank recovery: eight of eight directions.
- Digits MLP, four regimes, three deterministic seeds.
- Constraint alignment: 9.04% baseline versus 64.81% constrained.
- Matched 5,040-example delta: +0.67 percentage points IID and +0.67 points synthetic noisy OOD.
- Final filtering tradeoff: +0.74 points noisy OOD and -1.19 points IID.
- Effective rank: 7.77 baseline versus 11.54 constrained; cleaner-spectrum criterion NOT MET.
- Leave-one-source-out rank-eight worst principal angle: 89.81 degrees; weakest included direction unstable.
- Rank nine after holdout: NOT IDENTIFIABLE.

## Evidence policy

The source paper's claims are attributed to the paper. Conceptual diagrams are labeled as reconstructions. JSON experiment records are self-hashed. The serialized PyTorch basis has a tested schema and SHA-256. Missing ViT, LLM, natural-shift, and diverse-source experiments remain missing.

## Cost economics (synthesis, not measurement)

Evidence boundary: the cost model is arithmetic extrapolation from stated dense-transformer assumptions. Literature results remain attributed to their papers. The confidence-weighted basis and amortized scorer are untested proposals.

For one 2,048-token scoring unit, F ≈ 2PT. Backward is approximated as 2F, so ordinary training is 3F. Offline scoring breaks even when cF + 3pF < 3F, or p < 1 - c/3.

| Scale | One forward F | Ordinary step 3F |
|---|---:|---:|
| 100M | 0.410 TFLOPs | 1.23 TFLOPs |
| 1B | 4.10 TFLOPs | 12.3 TFLOPs |
| 7B | 28.7 TFLOPs | 86.0 TFLOPs |

Top three implementations:

1. Online last-block alignment gate: reuse the forward, backpropagate only to a late layer, contract activation-error factors against the basis, and skip the remaining backward for rejected examples.
2. Proxy scoring: recover an architecture-specific basis in a 25–100M proxy and test whether its ranking transfers to the target.
3. Amortized scorer: predict alignment from activations or embeddings, refreshed with 2% sparse exact labels every 250 steps.

Pre-registered kill criteria:

- Online last-block gate: kill below Spearman rho 0.50, below 40% top-20 recall, or when a 40% drop saves no measured GPU time.
- Proxy scoring: kill below median rho 0.40, below 35% top-20 recall, or when selection fails to beat random at two keep fractions.
- Amortized scorer: kill below rolling rho 0.60, above 0.10 correlation drift, or above 10% of the backward FLOPs it saves.
- Shared basis: kill any direction whose 95% bootstrap upper confidence bound on LOSO angle exceeds 30 degrees.

## Information-theoretic proposals (not measured)

SDEF (Subspace-Differential Entropy Filtering): score example i by delta-H = H(P,Q_UWS) - H(P,Q_full), the excess bit-cost of predicting it with weights restricted to the universal basis; delta-H is never negative. Typology: low-full/low-UWS is core structure, keep. Low-full/high-UWS is memorized-niche, drop. High-full/high-UWS is pure noise, drop. Moderate/moderate is novel, sample. This is a function-space view of the existing gradient-alignment score, not a replacement for it. Kill if delta-H ranking correlates with the existing gradient-alignment score at rho below 0.50, below 40% top-20 overlap with it, or the second forward pass pushes total scoring cost past the full-gradient baseline of 3F with no accuracy gain over the gradient-only score.

SSER (Spectral Shannon Entropy Regularization): normalize each layer's singular values to p_i = sigma_i / sum(sigma_j) and add lambda times the Shannon entropy of p to the training loss, penalizing spectral spread directly. Motivated by the cleaner-spectrum criterion marked NOT MET above: effective rank rose under constraint (7.77 to 11.54), and full-spectrum Shannon entropy rose too (3.71 to 4.18 bits, computed from the same singular-value records). Kill if no lambda in the sweep reaches effective rank at or below the unconstrained baseline of 7.77 without more than 0.5 percentage points of IID accuracy loss on the digits setup.

CEGO: the existing hard-constraint regime already run on this page -- project most of each update back into the recovered basis -- is a version of Cross-Entropy Gradient Orthogonalization from the literature. Naming the term adds no new mechanism; the regime, and its NOT MET spectral side effect, were already measured above.

Search-bounded novelty claim: As of July 2026, no published study we found values pretraining examples by alignment with a cross-model weight basis.