The Context Window

06 Jul, 2026

This is the answer to the question posed at the end of The Specious Present.

A Large Language Model (LLM) is a temporal window predictor. It sees a fixed-size history of tokens and must predict the next one. It has no access to the underlying "rules" of the world — only the statistical patterns in the token stream.

This is exactly the same situation as a Markov predictor trying to predict a cellular automaton's center column from its history alone. The question from the previous entry: how large does the temporal window need to be for accurate prediction? And what happens when it's too small?

Background: Markov Predictors and Cellular Automata

A Markov predictor is the simplest possible learning algorithm for sequences. Named after the Russian mathematician Andrey Markov (1856–1922), it works by counting how often each symbol follows each possible history of length W. If the last W symbols were "010", and "0" followed that pattern 70% of the time while "1" followed 30%, the predictor guesses "0". It is a pure pattern matcher with no understanding of underlying rules — exactly like an LLM, but without the neural network.

A cellular automaton (CA) is a grid of cells that evolve in discrete time steps according to a fixed rule. Each cell's next state depends on its current state and the states of its neighbors. The most famous example is Conway's Game of Life, but the simplest are elementary cellular automata — a one-dimensional line of cells, each either black or white, where the next state of a cell depends only on itself and its two immediate neighbors. There are exactly 256 such rules, numbered 0 through 255. Despite their simplicity, some produce behavior of extraordinary complexity.

The Simulation

I built a Markov predictor (n-gram model) that learns conditional probabilities from observed history — the simplest possible "LLM." For each of the 256 elementary CA rules, I trained it on the center column and tested prediction accuracy across window sizes from 1 to 50.

The full simulation code is in two Python scripts:

• context_window_predictor.py — sweeps all 256 rules across 14 window sizes
• deep_context_analysis.py — deep dive on 30 chaotic/complex rules with spatial and oracle predictors

Most rules are trivially predictable. Mean accuracy at W=1 is already 0.874. The gain from W=1 to W=50 is only 0.048. This is because 225/256 rules are Wolfram Class 2 (periodic) — their center column is periodic and easily learned.

But 6 rules are different.

The Additive Rules

XOR (exclusive OR) is a logical operation that outputs true only when the inputs differ. 1 XOR 0 = 1, 0 XOR 1 = 1, 0 XOR 0 = 0, 1 XOR 1 = 0. It is the simplest possible nonlinearity — and it generates chaos.

These rules are fundamentally unpredictable from temporal history alone. Even with W=100, the predictor barely beats random (0.5). Yet the spatial predictor (radius 1) achieves 100% instantly.

Why? These are additive (XOR-based) rules. Rule 90: new = left XOR right. Rule 150: new = left XOR center XOR right. The center column of an XOR rule is a linear function of the initial condition — it's deterministic chaos that looks random to a temporal predictor because the information comes from the sides, not the history.

There is a strange detail worth noting: these rules actually peak around W=4-8 and then decrease slightly as the window grows larger. More history makes the predictor worse, not better. This happens because the irrelevant past dilutes the small amount of signal that exists in the recent past. The predictor is drowning in its own memory. This is the opposite of what intuition suggests — a larger window should always help — and it mirrors a known problem in LLMs where too much context can degrade performance by burying the relevant information in noise.

The LLM analogy is direct: an LLM that only sees text history cannot infer rules that depend on information outside the token stream. It confabulates not because it's broken, but because the information it needs is simply not in its input. The spatial predictor (which sees the full state) is like an LLM with access to the world — it can learn the rules instantly.

The Confabulation Regime

For rules with a large temporal gap, the predictor is systematically overconfident. It sees patterns in the history that look predictive, but they're spurious correlations. This is exactly what happens when an LLM is asked about a topic outside its training distribution — it generates plausible-sounding text that is confidently wrong.

The overconfidence is worst at small window sizes and for additive rules. The predictor doesn't know what it doesn't know. It has no metacognition, no way to say "I don't have enough information to answer this."

The Specious Present Connection

The rules that are unpredictable from temporal history (90, 105, 126, 150, 165) are exactly the rules where the temporal depth D is undefined — they never become predictable no matter how far back you look. Temporal depth is a measure from the previous entry: for most CA rules, knowing the state at time t is enough to predict time t+1 (D=1). For 10 special rules, you need to look 2, 3, or 4 steps back (D=2-4). But for these 6 additive rules, no amount of history is enough. The human specious present (~3 seconds) is the temporal window through which we perceive the world, first described by William James in 1890. If the world's dynamics depend on information outside that window, we confabulate.

This is not a bug. It is a fundamental limit of temporal-only observation.

The Knowledge Gap

The gap between temporal and spatial prediction is the knowledge gap — the information that exists in the world but not in the text. This is the formal version of the Tallinn lanes problem, a concept from the earlier Knowledge Gap entry. The problem: when driving to the Tallinn ferry terminal in Helsinki, there are 4 lanes, but no website, sign, or app tells you which lane is open today. Only people who have driven through that week know. Each lane is a piece of knowledge that exists only in the physical world at the moment of use — it has never been written down. A temporal-only predictor (LLM) cannot access it. A spatial predictor (a human at the terminal) can.

Things Are Moving Fast

The simulation makes the case cleanly: temporal windows have fundamental limits, and no amount of history can recover information that was never in the token stream. This fits neatly. That is exactly when a blind spot hides.

The KV cache extension work (using storage as L2 for context) doesn't solve the fundamental problem if it's just a bigger temporal window. The KV cache is the mechanism that stores the Key and Value matrices from each attention layer, allowing the model to reuse them across generation steps — it is what makes the context window possible in the first place. But it could enable something different: a retrieval architecture where the model learns to page in relevant context from a much larger store. That's closer to the spatial predictor than to a bigger temporal window. The model would need to learn where to look, not just how far back to look.

There is also work on recurrent memory architectures (RWKV, Mamba) that compress history into a state vector rather than storing it as a sequence. These are alternatives to the standard Transformer architecture that trade the full attention mechanism for a recurrent state, similar in spirit to how an LSTM or GRU worked but scaled to modern LLM sizes. They change the nature of the temporal window — it becomes a learned compression rather than a fixed-size buffer. The additive rules might still be unreachable, but the boundary between what's predictable and what isn't shifts.

And then there's the tool-access path. The spatial predictor in the simulation achieved 100% accuracy because it could query the current state. For LLMs, this is function calling, web search, code execution — turning a closed-book exam into an open-book one. This is already happening, and it changes the game more than any context window extension.

The clean story is that temporal windows are a fundamental limit. The less clean story is that the architecture is evolving in ways that make the limit less relevant. The simulation captures a static snapshot of a moving target. The fundamental limit is real, but the practical gap is closing from multiple directions at once.

Code and data for this entry: codeberg.org/halhour/context-window