CONTEXT JAMMING / LANGUAGE MODELING

BEYOND
FIXED LENGTH

How segment recurrence and relative position let self-attention remember farther than any single segment

The paper’s central move is to enable pure self-attention models to capture dependencies longer than any fixed segment by introducing segment-level recurrence that re-uses cached hidden states across segments, made possible only by replacing absolute positional encodings with a relative scheme that preserves temporal coherence.

ENTER THE CUT Read the paper (arXiv:1901.02860v3) ↗

VANILLA TRANSFORMER Same corpus. The reversal is whether hidden states can cross the segment cut.

01 · Fixed segments

The Transformer that could only see its own window

Self-attention can, in principle, connect any pair of positions. In practice, training a deep Transformer language model on an entire corpus as one sequence is infeasible. The standard workaround — used by Al-Rfou et al. (2018) for character LMs — was to chop the corpus into fixed-length segments of a few hundred tokens and train only inside each segment.

Inside a segment of length L, every token can attend to every earlier token. Across the cut, nothing flows: not in the forward pass, not in the backward pass. The longest dependency the model can ever learn is bounded by L — the size of the window you chose before training began.

Autoregressive language modelingP(x) = ∏_t P(x_t | x_<t)

The factorization is standard. The fight is over how much left context the network can actually use.

Show context

§3: each conditional is parameterized by a neural encoder of the left context. The architectural question is how much of x_<t the encoder is allowed to see.

02 · Context fragmentation

Why the first tokens of every segment are almost blind

Segments are usually cut by character or token count for efficiency — not at sentence or paragraph boundaries. The first positions in each chunk therefore start prediction with almost no left context, even when the previous sentence is sitting one index earlier in the original text.

The paper calls this context fragmentation. It wastes capacity on systematically under-conditioned early tokens and produces an optimization signal that is partly an artifact of the cut. Evaluation can paper over the problem by sliding a window one token at a time and re-encoding from scratch — but that procedure is extremely expensive (Fig. 1b, §3.1).

03 · Recurrence without recurrence

Cache the hidden states, stop the gradient, keep going

Transformer-XL’s first technical move: when segment τ+1 begins, do not throw away segment τ. Cache its hidden states, stop the gradient through them, and concatenate them as extended context for keys and values.

Segment-level state reuse (§3.2)h̃_{τ+1}^{n-1} = [ SG(h_τ^{n-1}) ∘ h_{τ+1}^{n-1} ]

The green paths in Fig. 2a: information can flow forward across the segment cut even though gradients cannot.

Show context

Queries come from the current segment only; keys and values are produced from the extended context h̃. SG(·) = stop-gradient. ∘ is concatenation along length.

CONCEPTUAL RECONSTRUCTION of Fig. 2b · L = 4, N layers selectable

Show memory pathsN layers3

Effective context length = N × L = 12 tokensWith L = 4 and N = 3, the paper’s schematic effective context is 12 tokens (Fig. 2b, §3.2). Memory paths visible.

Interactive: three segments of length L = 4. Toggle memory paths; vary layer count N. Effective schematic context = N × L (Fig. 2b).

04 · O(N × L)

Effective context becomes linear in the number of layers

Recurrence does not create same-layer loops the way an RNN does. Each step, the dependence of layer n on the previous segment lands one layer lower. Stacked over N layers, the longest path through cached states therefore scales as O(N × L) — linear in depth and segment length — visualized as the shaded region in Fig. 2b.

Schematic effective contextO(N × L)

Longer than any single segment; still structured by the stop-gradient cuts between segments.

Show context

§3.2: “the largest possible dependency length grows linearly w.r.t. the number of layers as well as the segment length.” This is an upper bound on the path length through the recurrence, not a claim that every token attends N×L steps with equal strength.

05 · Absolute position fails

Why you cannot simply reuse states with the original positional encodings

If you keep absolute positional encodings and reuse hidden states, both segment τ and segment τ+1 receive the same vectors U_1 … U_L. The model then cannot tell x_τ,j from x_τ+1,j by position — “resulting in a sheer performance loss” (§3.3).

Absolute encoding under reuse (failure mode)

h_{τ+1} = f(h_τ, E_{s_{τ+1}} + U_{1:L})
h_τ = f(h_{τ−1}, E_{s_τ} + U_{1:L})

Same U on both segments → temporal collision under state reuse.

ILLUSTRATIVE MODEL of §3.3 failure mode

Temporal confusion: U₁:L reused on both segments§3.3: both segments receive the same absolute encodings, so x_τ,j and x_τ+1,j look identical in position.

06 · Relative positional encodings

Inject only the distance i − j into the attention score

The fix is not a better absolute table. It is to stop putting absolute indices into the residual stream and instead inject relative distances into the attention logits — so a query only needs to know how far each key is, not which absolute index it carried when first cached.

Four-term relative attention (§3.3)

A^{rel}_{i,j} = (a) E_{x_i}^T W_q^T W_{k,E} E_{x_j}
         + (b) E_{x_i}^T W_q^T W_{k,R} R_{i−j}
         + (c) u^T W_{k,E} E_{x_j}
         + (d) v^T W_{k,R} R_{i−j}

(a) content addressing · (b) content-dependent positional bias · (c) global content bias · (d) global positional bias.

Show context

R is a fixed sinusoid matrix (no learned absolute U). u, v are trainable global bias vectors. W_{k,E} and W_{k,R} are separate key projections for content and location. Terms (c)(d) are absent from Shaw et al. (2018).

ILLUSTRATIVE MODEL of the four-term decomposition (§3.3)

Toggle each A_rel term. Matrix is synthetic softmax over causal positions.

Active terms: a, b, c, d(a) content · (b) content×relative position · (c) global content bias · (d) global positional bias. Shaw et al. keep only (a)(b); Transformer-XL keeps all four with sinusoid R.

Because R uses the sinusoid inductive bias, a model trained at one memory length can generalize to longer memories at evaluation — the property Table 6 measures and §3.3 highlights as an advantage over merging W_k R into a single learned matrix.

07 · One layer, one head

The exact recurrence + relative attention step

Put both pieces together for a single layer and head: extend the context with stop-gradient memory, form Q from the current segment and K,V from the extended sequence, score with A_rel, apply a causal mask, then residual + feed-forward as usual.

CONCEPTUAL RECONSTRUCTION of §3.2–3.3 single-layer step

Exact schematic of the N-layer procedure at the end of §3.3 (single head). Multi-head and batching omitted for clarity.

Layer update (schematic, §3.3)h_τ^n = Positionwise-FFN( LayerNorm( Linear(a_τ^n) + h_τ^{n-1} ) )

Multi-head attention is the product of this step across heads; the page omits head concatenation for readability.

Show context

With a_τ^n = Masked-Softmax(A_τ^n) v_τ^n and h_τ^0 ≔ word embeddings of the segment.

08 · Memory length M

Train at 128–384, evaluate at 640–3,800

Memory length M is how many past hidden states you keep. Training typically sets M to the segment length (cheap BPTT). Evaluation can raise M — sometimes by several times — because relative encodings still make sense for distances never seen as absolute training indices.

PAPER-DERIVED VALUES (Table 6; interpolate only when necessary and label it)

TRAINING MEMORY128EVAL MEMORY640WIKITEXT PPL23.09

Evaluation memory length M640

PPL 23.09 · exact Table 6 pointTable 6 (151M, full loss, our encodings): PPL improves from 23.43 at training length to 23.09 at attention length 640. Relative sinusoid encodings enable evaluation memory longer than training memory — absolute encodings do not generalize the same way.

Also §4.1: WikiText-103 train attn 384 → eval 1,600; enwik8 train 784 → eval 3,800.

Setting	Train attention / memory	Eval attention / memory	Source
WikiText-103 (standard, §4.1)	384	1,600	PAPER-DERIVED
enwik8 large (§4.1)	784	3,800	PAPER-DERIVED
Table 6 ablation (151M, best)	~128 BPTT	up to 640	PAPER-DERIVED

09 · Numbers that moved the field

0.99 bpc, 18.3 perplexity, 1,800× faster

Architecture claims only stick when the leaderboard moves. Transformer-XL set new single-model results on word- and character-level benchmarks and simultaneously made evaluation dramatically cheaper by not recomputing every shifted window.

ENWIK80.99 bpcWIKITEXT-10318.3 PPLEVAL SPEEDUPup to 1,874×

PAPER-DERIVED VALUES from Table 9

Attention length3,800

ATTN LEN3,800VANILLA SLOWER BY1,874xMAX IN PAPER1,874x

Table 9 reports how much slower Al-Rfou et al. (2018) is than Transformer-XL at the same attention length (per-token time, one GPU). Speedup comes from reusing cached segment states instead of recomputing every shifted window from scratch (§3.1–3.2, §4.5).

Relative effective context length

Using RECL (Table 8), Transformer-XL models about 900-word dependency at r = 0.1 — roughly 80% longer than the RNN baselines in its group and 450% longer than vanilla Transformer (128).

PAPER-DERIVED VALUES from Table 8

At r = 0.1: XL RECL 900 vs QRNN 500 vs LSTM 400 vs Transformer 128Abstract claim: RECL is 80% longer than RNNs and 450% longer than vanilla Transformer (at r = 0.1: 900 vs 500 and 128). RECL is defined on a model group; see Appendix C.

Evidence label: PAPER-DERIVED VALUES. This page does not retrain models; charts read Table 8 and Table 9 directly. RECL is a group metric with hardness parameter r — not raw ECL.

10 · What still does not work

Gradient still does not cross segments; memory still costs GPU RAM

Segment recurrence is not free unlimited memory. Gradients stop at the segment boundary (SG). Cached states cost activation memory proportional to M × layers × dimension. Very long evaluation contexts still hit hardware limits even when the positional scheme generalizes.

STILL TRUE AFTER TRANSFORMER-XL

No gradient through cached segments — truncated credit assignment (§3.2).
Effective path length is O(N × L), not infinite context.
Memory M is bounded by GPU RAM; more history is more state.
Relative encoding is a specific four-term sinusoid form, not an arbitrary RoPE.

WHAT THE ARCHITECTURE FIXED

Cross-segment information in the forward pass.
Context fragmentation at segment starts (Table 7 on One Billion Word).
Evaluation recomputation cost (Table 9).
Train/eval memory mismatch under absolute encodings (Table 6).

AUTHOR INTERPRETATION

First pure self-attention LM to beat strong RNNs on both character and word benchmarks (Introduction / Conclusions).
Long coherent generation from WikiText-103-scale data is presented as qualitative evidence (Appendix E) — not a controlled human eval.

11 · STRUCTURAL ANALOGY, NOT IDENTITY

From segment memory to agentic long-horizon memory

STRUCTURAL ANALOGY, NOT IDENTITY

Transformer-XL’s 2019 design — cache past states, stop the gradient, bias attention by relative distance — is a direct ancestor of today’s long-context engineering. The visual and engineering rhyme is real. The mathematical and product identity is not.

PAPERSegment-level cached hidden states↔MODERN ANALOGUEModern KV-cache / sliding-window attention caches

PAPERRelative positional encoding that generalizes beyond train length↔MODERN ANALOGUERoPE / ALiBi-style methods that extrapolate or bias by distance

PAPERStop-gradient across segments↔MODERN ANALOGUETruncated BPTT / gradient-checkpointing trade-offs

Where the analogy breaks

Transformer-XL still truncates gradients at segment boundaries.
Its relative encoding is a specific four-term sinusoid form.
2019 speedup numbers do not transfer directly to modern kernels.
The paper never claims an agentic or multi-turn setting.

Falsifiable research questions

Does increasing cached segments in a modern KV-cache produce the same linear effective-context growth?
At what memory length does Transformer-XL-style relative bias saturate on current long-context benchmarks?
Can pure stop-gradient segment recurrence still compete when segment length is made adaptive?

12 · Epistemic ledger

What the paper actually establishes

ESTABLISHED BY THE PAPER

Segment-level recurrence + relative encodings enable cross-segment context without absolute-position collision (§3.2–3.3).
New SoTA LM numbers including 0.99 bpc enwik8 and 18.3 PPL WikiText-103 (Tables 1–2).
RECL up to 900 words at r = 0.1; ~80% / ~450% longer than RNN / vanilla Transformer groups (Table 8).
Up to 1,874× faster evaluation vs vanilla recompute (Table 9).
Ablations: both recurrence and the proposed encoding matter; eval memory > train memory helps only with their relative scheme (Table 6).

AUTHOR INTERPRETATION

First self-attention model to substantially beat RNNs on both character- and word-level LM (Introduction).
Qualitative long-form generation (Appendix E) as evidence of long-range coherence.
Suggested applications to generation, unsupervised features, image and speech (Conclusions) — not evaluated here.

CONTEXT JAMMING EXTENSION

Structural analogy to modern KV-caches, RoPE/ALiBi, and truncated credit assignment — labeled NOT IDENTITY.
Falsifiable questions listed above; none are results of the 2019 paper.

13 · Field guide

A compact glossary

Fixed-length context

Training and evaluating a Transformer only inside a pre-chosen segment of length L, with no information from earlier segments (§3.1).

Context fragmentation

The problem that the first tokens of each fixed chunk lack left context because segments are cut without respecting sentence or discourse boundaries (§1, §3.1).

Segment-level recurrence

Caching the previous segment’s hidden states and concatenating them (with stop-gradient) as extended context for the next segment (§3.2).

Stop-gradient (SG)

A detach operation: previous-segment states inform the forward pass but do not receive gradients, so backprop stays within the current segment (§3.2).

Memory length M

How many past hidden states are cached. During training M is typically the segment length; at evaluation M can be increased (§3.2, Table 6).

Effective context O(N × L)

Because recurrence shifts one layer down per segment, the longest dependency path scales with layers × segment length, not just L (Fig. 2b).

Absolute positional encoding

Fixed vectors U_i added by absolute index inside a segment. Reusing states with the same U_1:L collides positions across segments (§3.3).

Relative positional encoding

Injecting only the distance i − j (via sinusoid R) into attention scores so temporal order stays coherent under state reuse (§3.3).

Four-term A_rel

Decomposition of relative attention into (a) content, (b) content-dependent position, (c) global content bias, (d) global positional bias (§3.3).

RECL

Relative Effective Context Length: a group-relative metric of how far context still helps, with hardness parameter r (Table 8, Appendix C).

bpc / perplexity

Bits per character (character LM) and word-level perplexity (word LM). Lower is better. Headline: 0.99 bpc enwik8, 18.3 PPL WikiText-103.

Vanilla Transformer LM

Here: Al-Rfou et al. (2018)–style fixed-segment training/evaluation without cross-segment memory (§3.1, Table 9 baseline).

14 · Source notes

The argument, pinned to the paper

Thesis & abstract results: Abstract; 80% / 450% RECL claims; 0.99 bpc; 18.3 PPL; 1,800+× evaluation speedup.
Fixed segments & context fragmentation: §1, §3.1; Fig. 1a–b (vanilla train/eval).
Segment-level recurrence, SG, O(N×L): §3.2; Fig. 2a–b; equations for h̃, q/k/v.
Absolute encoding failure: §3.3 opening paragraphs; schematic with U_1:L on consecutive segments.
Relative four-term A_rel: §3.3; comparison to Shaw et al. (2018); sinusoid R; efficient computation in Appendix B.
Full layer procedure: end of §3.3 (N-layer single-head summary).
Main results: Tables 1–5 (WikiText-103, enwik8, text8, One Billion Word, PTB).
Ablations & memory generalization: Table 6 (WikiText-103); Table 7 (One Billion Word fragmentation control).
RECL: §4.3, Table 8, Appendix C.
Evaluation speed: §4.5, Table 9 (800 / 1,800 / 2,800 / 3,800 → 363× … 1,874×).
Generation samples: Appendix E (qualitative).

Dai, Zihang, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc V. Le, and Ruslan Salakhutdinov. “Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context.” arXiv:1901.02860v3 [cs.LG], 2 June 2019. ACL 2019.

Open the primary source ↗

← BACK TO EXPLAINERS RETURN TO CONTEXT JAMMING