The RNN
Compresses the sequence into a fixed-size state. Cost stays linear; capacity does not grow with the sequence.
CONTEXT JAMMING / MEMORY CACHING
ATTENTION WAS ALWAYS A CACHE
How checkpointing a recurrent memory turned the choice between an RNN and a Transformer into a single tunable coordinate.
For a decade the field treated fixed-memory recurrence and growing-memory attention as opposed designs with opposed cost curves. Behrouz and colleagues propose that the opposition is partly an artifact of never naming the axis between them—and that the axis is how often a memory checkpoint is saved (§3).
An RNN and a Transformer are not two architectures. They are two settings of N.
01 · The cast
Three ways to hold the past
The difference is not whether the model has memory. It is how finely the past is retained (§3, Eqs. 4–5).
Attention
Caches every past token. Access is direct, but compute becomes quadratic and inference KV memory grows.
Memory Caching
Compresses each segment into a learned state, then caches those states. Segment count sets the position between the endpoints.
02 · Coordinate reversal
The architecture becomes a number
Split a length-L sequence into N segments and retrieve from the online memory plus the cached segment memories. Update remains O(L); retrieval adds O(N) work per token, giving O(NL), with 1 ≤ N ≤ L (§3.1, “Memory Complexity”).
SEGMENT SIZE C8CACHED MEMORIES AT END7RETRIEVAL OPS / TOKEN∝ 8REGIMEMEMORY CACHING
03 · Checkpoints or compressors
Two readings of the same cache
Optimization reading
Tokens are training samples and the memory state acts like parameters. Each segment begins from the previous segment’s last state, preserving an optimizer checkpoint against forgetting (§3.4).
Compression reading
Each cached memory is a self-contained representative of one segment. Independent initialization prevents one segment’s state from interfering with another (§3.4).
The authors report that each choice has advantages and disadvantages; §5.6 does not establish a universal winner.
04 · The simplest aggregation
Add the past back in
Residual Memory applies the same query to the online memory and every cached state, then sums the outputs (Eq. 7). For a strictly linear matrix memory, that construction collapses: the matrices can be pre-summed into one fixed-size memory (Eqs. 12–13).
05 · Averaging the memories
Memory Soup
Memory Soup interpolates cached-memory parameters first and applies the query once to the resulting input-dependent memory M* (Eqs. 14–15). For a linear memory it is mathematically equivalent to GRM. With a deep nonlinear memory such as DLA or Titans, parameter interpolation constructs a new retrieval function and the equivalence breaks (§3.2).
Linearity makes souping weights before the query identical to summing weighted outputs.
06 · Routing the cache
Sparse Selective Caching
SSC scores cached segments by the same connector-to-mean-pooled-content similarity, chooses Top-k, and retrieves only those memories plus the online one (Eqs. 16–17). Segment summaries can be precomputed, scoring is parallelizable, and only selected memories must be loaded onto the accelerator (§3.3).
Each token activates a subset of parameters for writing and a larger subset for reading: a sparse unified memory whose block granularity is set by segment size. — paraphrase of §3.3, “Effective Memory”
ACTIVE MEMORIES: 4 of 93 cached segments plus the online memory are active.
07 · The collapse
Set N to L and attention falls out
Set segment size to one and use a value-less vector memory: each state stores one projected token (Eq. 18). Cache those states, softmax-normalize γ (Eq. 19), and reparameterize each state as v′ᵢ. The remaining expression is a softmax-weighted sum of value vectors gated by an input projection—the gated global attention block (Eq. 20).
STEP 1 · ONE TOKEN PER SEGMENT
M₁⁽ᵗ⁾ = bK + xₜWKEach step changes one term in the construction.
The neighboring hybrid result is also limited but revealing: a vector-memory compressor followed by global attention is equivalent to checkpoint-style MC with segment size one in the paper’s oversimplified construction (Eqs. 21–24). With normalization and feed-forward layers, expressivity can differ (§4.1).
08 · Where to cut
Constant, or logarithmic
Equal segments of size C=L/N cost O(p·L²/C). Logarithmic segmentation writes L in binary and uses the powers of two corresponding to set bits; 37=(100101)₂ becomes 32, 4, 1, with O(p·L log L) cost (§4.2, Fig. 3).
The cheaper scheme loses resolution in the distant past. At 16K, a single early memory can be forced to compress 8K tokens; §5.2 uses this to explain Log-Linear++ weakness on long-context NIAH. Constant-size variants outperform it in Tables 1–3.
SEGMENTS N3LARGEST SEGMENT32RETRIEVAL OPS∝ 3
09 · The evidence
What the tables actually show
| Model | Metric | Base | + GRM | Locator |
|---|---|---|---|---|
| Titans (LMM) | 8-task average | 56.82 | 58.33 | Table 1 |
| Titans (LMM) | Wikitext perplexity ↓ | 15.60 | 15.37 | Table 1 |
| DLA | 8-task average | 53.72 | 55.96 | Table 1 |
| Titans (LMM) | S-NIAH-3 @ 16K | 21.2 | 32.2 | Table 2 |
| DLA | S-NIAH-3 @ 16K | 4.0 | 18.2 | Table 2 |
| DLA | Retrieval average | 30.51 | 38.03 | Table 3 |
| Titans (LMM) | Retrieval average | 31.75 | 40.50 | Table 3 |
| Transformer | Retrieval average | 41.00 | — | Table 3 |
Values transcribed verbatim. No rounding, interpolation, or recomputation.
Across the reported variants the recurring ordering is GRM ≥ Memory Soup ≥ SSC > Log-Linear++ > base, while SSC offers the strongest efficiency trade-off. Figure 4 places MC variants between Transformers and RNNs in throughput and reports that the efficiency gap widens with context.
MC can also be applied after pre-training by caching inference-time states and decoding with an unweighted moving average. The authors say this meaningfully improves length extrapolation, but provide no numeric result for that claim (§4.3, final paragraph).
10 · Where it still loses
The gap narrows; it does not close
- Transformers still win in-context recall. The retrieval average is 41.00 for the Transformer and 40.50 for Titans+GRM. The paper says MC is competitive and closes the recurrent gap—not that it beats attention (Abstract; Table 3; §5.3).
- LongBench is mixed, not uniform. Against base Titans, GRM regresses on GovReport 10.5→8.4, MultiNews 11.8→3.1, and TREC 37.1→14.8. Table 4’s individual columns are therefore narrower than §5.4’s sentence that all MC variants provide gains.
- Table 5 contains placeholders. “Shared u and q” shows 00.0 / 00.0 / 00.0 for GRM and SSC. PLACEHOLDER IN v1 — NOT A REPORTED RESULT
- Academic scale. The experiments cover 760M and 1.3B parameters trained on 30B and 100B tokens. They do not establish frontier-scale behavior (§5.1; Table 1).
- The largest gains concentrate in retrieval. Language-modeling and commonsense gains are modest relative to NIAH and recall deltas (Tables 1–3). “Better language models” would be too broad a headline.
11 · Context Jamming extension
The coordinate you didn't know you were choosing
EDITORIAL EXTENSION · NOT CLAIMED BY THE PAPER
The conceptual reversal rhymes with Kinematic Space: there, a place becomes an interval coordinate; here, an architecture choice becomes segment count. This is an editorial analogy, not a citation or mathematical connection.
PAPERSegment count N↔PRACTITIONER READINGPosition on the cost/recall curve
PAPERCached memory state↔PRACTITIONER READINGCompressed checkpoint of a span
PAPEROnline memory↔PRACTITIONER READINGThe state still being written
PAPERContext-dependent γ↔PRACTITIONER READINGRetrieval weighted by content, not position alone
PAPERTop-k router↔PRACTITIONER READINGSparse retrieval over compressed history
PAPERCheckpoint vs compressor↔PRACTITIONER READINGContinuity versus isolation
PAPERPost-training MC↔PRACTITIONER READINGLength extrapolation as a deployment-time change
Two lineages, kept in bounds
Transformer-XL (N° 15) is structural affinity plus documented lineage: it caches previous-segment hidden states and appears in MC’s related work. MC expands that instinct into learned, gated, routed retrieval over past segment memories.
Kinematic Space (N° 03) is an editorial analogy only. Both replace a coordinate system and index by position and scale; there is no citation, shared authorship, or formal relationship.
Three questions an experiment could answer
Does the optimal N vary by task family?
- Measurable object
- Best N on LM, retrieval, and NIAH.
- Possible experiment
- Sweep fixed N at matched compute.
- Falsifying result
- One N dominates across tasks.
- Caveat
- Training and inference N may interact.
Do routers select semantic spans or recent ones?
- Measurable object
- Selected segment content and distance.
- Possible experiment
- Control semantic relevance against recency.
- Falsifying result
- Selections reduce to position bias.
- Caveat
- Mean pooling may hide token-level cues.
Does post-training MC preserve calibration?
- Measurable object
- Calibration error beyond training length.
- Possible experiment
- Compare base and moving-average MC decoding.
- Falsifying result
- Recall rises while calibration collapses.
- Caveat
- Post-training MC has no numeric result in v1.
12 · Epistemic ledger
What the paper actually establishes
- MC improves the tested SWLA, DLA, and Titans bases across the paper’s benchmark families at two academic scales (Tables 1–4).
- Constant segmentation beats Log-Linear++ on the reported recall comparisons (Tables 1–3).
- GRM is strongest overall; SSC trades some quality for efficiency (Tables 1–3; Fig. 4).
- Context-dependent gating is the largest retrieval contributor in the completed Table 5 rows.
- Memory Soup equals GRM for linear memory and diverges for nonlinear memory (§3.2).
- The simplified segment-size-one construction recovers gated global attention (§4.1).
- Simplified hybrid attention-plus-recurrence can be read as checkpoint-style MC (§4.1).
- Cached states can be understood as optimizer checkpoints (§3.4).
- N as a deployment coordinate rather than an architectural commitment.
- The Transformer-XL lineage reading and kinematic-space rhyme.
- Any application to agent context management.
13 · Field guide
A compact glossary
Online memory
The current segment’s memory state, still being updated.
Cached memory
The final learned memory state saved from a completed segment.
Segment
A contiguous chunk compressed into one memory state.
Residual Memory
The online memory output plus the sum of cached-memory outputs (Eq. 7).
Gated Residual Memory (GRM)
Residual retrieval with input- and segment-dependent weights (Eqs. 8–10).
Memory Soup
Input-dependent interpolation of memory parameters before a query is applied (Eqs. 14–15).
Sparse Selective Caching (SSC)
Top-k retrieval from cached memories chosen by a contextual router (Eqs. 16–17).
Router
The scoring rule that selects cached segments relevant to a query.
Top-k selection
Keeping only the k highest-scoring cached memories.
Mean pooling
A segment summary formed by averaging its token representations.
Linear attention
A recurrent matrix-valued memory with linear-time updates.
Deep Linear Attention (DLA)
A recurrent architecture whose memory is a learned deep module.
Titans
A test-time memorization architecture used as one MC base model.
SWLA
Sliding Window Linear Attention, another recurrent update rule tested with MC.
Log-linear attention
A hierarchical recurrent design using logarithmically many states.
Fenwick tree
The binary-indexed data structure motivating log-linear segmentation.
Attentional bias
The objective used to update an associative memory from keys and values.
Test-time memorization
The view that a model trains an internal memory during its forward pass.
Associative memory
A state updated to map keys to values or learned representations.
Value-less memory
A memory whose mapping depends only on keys.
Disentangler-free compression
Compression without a separate operation that explicitly removes local correlation.
Bond/state dimension
The width limiting a compressed memory state’s representational capacity.
KV-caching
Storing attention keys and values for raw past positions; not MC’s compressed segment states.
NIAH
Needle-In-A-Haystack retrieval tests.
MQAR
Multi-Query Associative Recall.
LongBench
A suite of long-context understanding tasks.
Perplexity
A language-model uncertainty metric; lower is better.
Length extrapolation
Using a model beyond the context lengths seen in training.
Weight souping
Averaging trained model parameters, the inspiration for Memory Soup.
Mixture of Experts
Sparse conditional routing across parameter blocks, the inspiration for SSC.
14 · Source notes
The argument, pinned to the paper
- Thesis and O(NL): Abstract; §1 Contributions; §3 opening.
- Segmentation and update rule: §3, Eqs. 4–5.
- Residual Memory and collapse: §3.1, Eqs. 6–7 and 12–13.
- GRM and content gate: §3.1, Eqs. 8–10.
- Memory Soup: §3.2, Eqs. 14–15.
- SSC router: §3.3, Eqs. 16–17; Fig. 2.
- Checkpoint/compressor fork: §3.4; §5.6.
- Attention recovery: §4.1, Eqs. 18–20; hybrid discussion following.
- Segmentation arithmetic: §4.2; Fig. 3.
- Applications: §4.3, Eqs. 26–36.
- LM and commonsense: Table 1.
- NIAH / retrieval / LongBench: Tables 2–4.
- Ablations / throughput / MQAR: Table 5; Figs. 4–5.
- Post-training MC: §4.3, final paragraph.
Behrouz, Ali, Zeman Li, Yuan Deng, Peilin Zhong, Meisam Razaviyayn, and Vahab Mirrokni. “Memory Caching: RNNs with Growing Memory.” arXiv:2602.24281v1, 27 February 2026. Google Research, Cornell University, and University of Southern California.
Open the primary source ↗