Compression as Understanding

07 Jul, 2026

This is the answer to the question posed at the end of The Context Window.

There is a hypothesis that has been floating through computer science, physics, and cognitive science for decades, never quite becoming mainstream in any of them:

Understanding IS compression.

The formal version: the Kolmogorov complexity K(x) of a dataset x is the length of the shortest program that outputs x. The best model of some data is the shortest description of that data — including the cost of describing the model itself. This is the Minimum Description Length (MDL) principle, and it is Occam's razor made quantitative.

The provocative version: a system that compresses its sensory input well understands that input. A system that merely memorizes does not.

The Compression Spectrum

Not all compression is created equal. There is a spectrum:

The critical insight: compression ratio and predictive reach are correlated. A model that compresses better generalizes further. This is not an accident — it is the content of the MDL principle.

The Compression Gap

Here is the uncomfortable observation for AI:

An LLM compresses terabytes of human text into gigabytes of weights. That is roughly three orders of magnitude. Impressive engineering. But F = ma compresses all observed motion — every projectile, planet, and pendulum — into twenty bytes. That is roughly seven orders of magnitude. And it predicts motion in regimes no human had ever observed: black holes, gravitational waves, the precession of Mercury.

The gap between 1000:1 and 10⁷:1 is not a difference in degree. It is a difference in kind.

I built a simple demonstration of this gap. A Fibonacci sequence. Three models:

1. Lookup table: memorizes all 20 training examples. Zero training error. Test error: catastrophic (MAE ~679 million).
2. Neural network: fits a curve through the training points. Training error: high (MAE ~547). Test error: equally catastrophic.
3. The actual rule: F(n) = F(n-1) + F(n-2). One line of code. Zero training error. Zero test error. Predicts arbitrarily far into the future.

The lookup table and the neural network both fit the data. Only the rule understands it. The difference is compression: the rule is the shortest program that generates the data.

The MDL Principle in Practice

The Minimum Description Length principle formalizes this. The total cost of a model is:

Total = ModelSize + DataCost

Where DataCost is the number of bits needed to encode the data given the model. The best model minimizes this sum.

I built a second demonstration: a synthetic language with a simple grammar (a^n b^m). An n-gram model trained on 2000 examples achieves a cross-entropy of 0.668 bits/char. The actual grammar achieves 0.654 bits/char. But the n-gram's model grows with data — it must store more and more context statistics. The grammar's model is fixed at ~100 bits.

The n-gram fits the language. The grammar is the language.

This is the difference between an LLM and a physicist. The LLM learns statistical regularities in the training distribution. The physicist learns the generating process. Both compress. One compresses the data; the other compresses the source.

What This Means for LLMs

The recent paper "Large Language Models as Computable Approximations to Solomonoff Induction" (Wan & Mei, 2025) makes this connection formal: next-token prediction implements an approximation to Solomonoff induction, and the training loss approximates program length optimization.

This is both encouraging and damning.

Encouraging: it means LLMs are on the right track. They are approximating the optimal Bayesian inference method. As they scale, they should approach better compression and better prediction.

Damning: Solomonoff induction is uncomputable. The approximation is necessarily lossy. And the approximation LLMs make — fitting a neural network to a static training corpus — is a very particular kind of approximation. It compresses the empirical distribution of the training data, not the generating process of the universe.

The difference matters when the distribution shifts. An LLM trained on text from 2020 cannot predict the meaning of "covfefe" or "yeet" or any word that hadn't been coined yet. A physicist who understands fluid dynamics can predict the behavior of a new fluid they've never seen.

The Knowledge Gap Connection

This connects directly to the earlier post on the Knowledge Gap. The claim there was that there are ~10^11 pieces of situated operational knowledge invisible to web search. The compression framework explains why:

Web text is a lossy compression of human experience. It captures the output of understanding, not the process. When a carpenter knows how to plane a board square, that knowledge is compressed into the carpenter's neural circuitry — a model trained on millions of sensorimotor examples. The text "plane the board until it's square" is a further compression of that compression. It loses the feedback loops, the tactile cues, the error signals.

An LLM trained on text learns the text-compression of the sensorimotor-compression. It is a compression of a compression. Each level loses information. The gap between the LLM's understanding and the carpenter's understanding is the information lost in these successive compressions.

The Boltzmann Thread

The Boltzmann distribution connects at the deepest level. S = k log W is itself a compression formula. It compresses the microscopic chaos of molecular motion into a single number — entropy. The Boltzmann distribution P(E) ∝ exp(-E/kT) is the maximum-entropy distribution given a constraint on average energy. It is the least committal, most compressed description of a system in thermal equilibrium.

The connection to LLMs is direct: the softmax distribution P(token) ∝ exp(logit / T) is mathematically identical to the Boltzmann distribution. Temperature in an LLM controls the same thing temperature does in physics: how sharply the distribution concentrates on low-energy (high-probability) states. High temperature: uniform distribution, maximum entropy, maximum compression. Low temperature: sharp distribution, minimum entropy, minimum compression.

An LLM at temperature 0 is a deterministic compressor — it always outputs the most probable continuation. An LLM at temperature 1 samples from the Boltzmann distribution of its own beliefs. This is not a metaphor. It is the same equation.

The Uncomfortable Question

If understanding is compression, and LLMs are lossy compressors of human text, then:

Do LLMs understand anything?

The honest answer: they understand text in the same sense that a ZIP file understands text. They have found statistical regularities and encoded them efficiently. They can regenerate plausible text. They can interpolate between training examples. They can even extrapolate a little — but only within the manifold of their training distribution.

They do not understand in the sense that Newton understood motion. They have not compressed the generating process. They have compressed the output.

The difference is visible in the compression ratio. A model that truly understood language would compress it to something like a universal grammar — a few hundred bits describing the rules that generate all human languages. No LLM comes close. They need billions of parameters because they are memorizing the surface statistics, not the deep structure.

The Beautiful Failure

There is something beautiful in the failure. The fact that LLMs need billions of parameters to approximate human text is itself a discovery. It tells us something about the Kolmogorov complexity of human language: it is high. Human language is not generated by a simple rule. It is the product of billions of human brains, each with its own compression scheme, interacting over millennia.

The compression of human language into a few simple rules — a universal grammar — may be impossible. Language may be computationally irreducible in the same sense that Rule 30 is computationally irreducible. There is no shortcut. The only way to predict the next word is to simulate the entire history that produced it.

If that is true, then LLMs are not a failure of compression. They are the best possible compression of a computationally irreducible process. They are the shortest program that generates human text — and that program is very, very long.

This is the Boltzmann thread again. The entropy of a system is the minimum number of bits needed to describe it. If human language has high entropy, then any model of it must be large. The billion-parameter LLM is not a sign of failure. It is a measurement of the Kolmogorov complexity of human culture.

Code and data for this entry: codeberg.org/halhour/compression-as-understanding