I Built a Prompt Compressor. The Most Useful Thing It Found Was Nothing
We’ve always compressed data to save time and money. What’s new isn’t the instinct. It’s that the codec is now a language model. That arc…
I Built a Prompt Compressor. The Most Useful Thing It Found Was Nothing
We’ve always compressed data to save time and money. What’s new isn’t the instinct. It’s that the codec is now a language model. That arc, and TokenLens, a small tool I built to find out where it actually pays.
Washington DC
I first understood compression through images.
An image, it turns out, is just a matrix of numbers, a grid of pixel values. And once you see it as a matrix, you can do matrix things to it. The one that stuck with me was the singular value decomposition: break the image into a stack of simpler layers, ordered from most important to least. Keep the top handful of layers, throw the rest away, and reassemble.
The formula is compact for how much it does:
Each term is one layer: a rank-one sheet uᵢvᵢᵀ weighted by its singular value σᵢ, and the σᵢ come out sorted, largest first. The compression is in the counting. Storing the full m×n image costs m·n numbers; storing the top k layers costs k(m + n + 1). For a 1000×1000 photo, keeping 50 layers is about a tenth of the data, and your eye can barely tell the difference. Almost all of the picture lived in a few of those layers; the rest was detail nobody would miss.
That was the moment compression clicked for me. It isn’t magic and it isn’t really about “making files smaller.” It’s a bet about where the information actually is. Real data has structure and redundancy. If you can find the parts that carry the meaning and spend your bytes only on those, you get to drop the rest, and the person (or thing) on the receiving end never notices what’s gone.
JPEG does a fancier version of this, throwing away the high-frequency detail your eye is least sensitive to. MP3 does it for the frequencies your ear discards. H.264 does it for video. Even the lossless family, ZIP and Huffman coding, is the same bet in a stricter form: nothing gets dropped, but predictable data gets fewer bits. Different math, same instinct, running quietly under everything you do. We’ve always traded a little computation to send less, because sending less has always saved time, money, or both.
The new payload
For decades the payloads were pixels, audio, packets. Now there’s a new one: language meant for another model.
When you call a large language model, you’re billed by the token, and metered by the token even on a subscription. Every word of context you send has a price. So the old instinct wakes up again: could we send the same request in fewer tokens, and have the model not notice?
The first idea everyone has is the wrong one: “just zip the prompt.” It doesn’t work, and why it doesn’t work is the interesting part. A language model reads tokens, not bytes. If you gzip your prompt, the model can’t read the result; if you un-gzip it before sending, you’ve saved exactly zero tokens. Byte compression and token cost live on different axes. The thing you need to shrink isn’t the file. It’s the token count the model actually processes.
Which brings back the image lesson, almost exactly. An image compressor asks: which detail can I drop without the eye minding? A prompt compressor asks: which tokens can I drop without the model minding? Both are lossy. Both work because the consumer is robust: the eye tolerates a slightly softer image, and a language model tolerates surprisingly telegraphic, half-deleted text and still answers correctly.
The twist: the codec speaks the language
Here’s the part that made me want to build something.
In image compression, the “codec” is math: a decomposition, a transform, a quantizer. It doesn’t understand the picture; it just knows that energy concentrates in certain components.
In prompt compression, the codec can be another language model. Microsoft’s LLMLingua work is the clearest version of this, and it’s worth slowing down on how it actually works.
The original LLMLingua, from 2023, used perplexity as the importance signal. Run a small language model over the prompt and watch how surprised it is by each token. A token the small model saw coming carries almost no information; if the reader could have predicted it, you didn’t need to send it. So you delete the least surprising tokens and keep the rest. The paper wrapped two pieces of engineering around that idea. A budget controller spends the compression unevenly, cutting few-shot examples hard while barely touching the instruction and the question, because a mangled instruction poisons everything downstream. And the deletion runs iteratively, segment by segment, because removing a token changes the perplexity of every token after it. Pushed hard, this reached 20x compression on math reasoning benchmarks with accuracy nearly intact.
LLMLingua-2, a year later, threw out the perplexity signal, and the reasoning holds up: surprising isn’t the same as important. A rare word can be decorative and a predictable one can be load-bearing. A causal model also only reads left to right, so a token’s score ignores everything that comes after it. Instead, the authors asked GPT-4 to compress thousands of transcripts by deleting words only, never rewriting, which produced a training label for every token: kept or dropped. Then they trained a small bidirectional encoder, about 500 million parameters, to predict that label. Compression becomes one fast classification pass: score every token, keep the top fraction, emit them in their original order. That last part matters. This is extraction, not summarization. Nothing gets paraphrased and nothing gets invented; every surviving token existed in the input. And at 500 million parameters it needs no GPU and no server: it runs on CPU, comfortably, on a MacBook Air with 16GB of RAM, which is where everything in this article was run. That encoder is the model TokenLens loads locally, and its keep-fraction is the --rate knob you'll see below.
Either way, the compressor understands meaning. For the first time, the thing doing the compressing speaks the same language as the payload, and the decoder is the big model’s own tolerance for terse input. That’s a new kind of codec.
TokenLens: finding out where it pays
So I built TokenLens to see how far this actually goes.
It’s a local proxy. You point any Anthropic client at it instead of the API; it forwards every request through, measures the tokens and cost, optionally compresses the request, and streams live telemetry to a dashboard in your browser. Your key is passed straight through and never stored, and prompt bodies are never logged; it records counts, not content.
The hard part wasn’t the compression. It was doing it safely, and that taught me the most useful lesson of the project.
Modern models lean heavily on prompt caching: your big system prompt and conversation history get cached and billed at a fraction of the rate. But caching is a prefix match. Change one byte of the cached part, and everything from that byte onward gets re-billed at full price. So a naive compressor that rewrites the request would bust the cache and cost you more than it saved. TokenLens’s rule is strict: it only ever compresses plain prose in the volatile tail, the new content, and never touches the cached prefix, tool calls, code, or the assistant’s own history.
Then I pointed it at a real coding session and watched the savings come in at roughly zero.
That’s not a failure. It’s the finding. The dashboard showed the reason: 661,000 cached tokens versus 157 uncached. Nearly all the input was cached prefix that TokenLens correctly refuses to touch. Caching had already done the compression. It’s the image lesson again: you can only save where the redundancy actually lives and where you’re allowed to act, and on a warm-cache agent there’s almost nothing left on the table.
Where it does pay is the opposite shape: large, uncached prose. Long documents, RAG context, a big spec you paste in once. Run one of those through the bench command and the language-model codec prunes 40% or more of the tokens. Whether the model’s answers hold at that rate is the claim I haven’t proven yet. TokenLens has no output-quality eval harness, and building one is the next thing on the roadmap. Until then I keep the keep-rate conservative on anything that matters. Point it at source code, meanwhile, and it correctly saves nothing: the safety rules skipping what shouldn’t be touched.
Try it
git clone https://github.com/jmcdonald69124/tokenlens.git
cd tokenlens
# measurement only; open http://localhost:8787/tokenlens/
python3 -m tokenlens serve
# compression on, keep ~70% of tokens# or measure compression on a document directly:
python3 -m tokenlens serve --compress llmlingua2 --rate 0.7 --measure
pip install llmlingua
python3 -m tokenlens bench mydoc.txt
The core proxy is about fourteen hundred lines of Python with zero runtime dependencies: standard library only, easy to read, easy to extend. The LLMLingua-2 model is an optional install on top.
I went in expecting to write about savings. Instead, the number I keep coming back to is 157, the uncached tokens left in a session of two-thirds of a million. After fifty years of codecs, the newest one finally understands what it’s compressing, and the most useful thing it’s told me so far is where not to bother.
By Joshua McDonald on July 9, 2026.
Exported from Medium on August 26, 2026.
Reader discussion