You Can’t Compress What You Can’t Grade
Adding an eval harness to TokenLens, and why it should have come before the compression work. A follow-up to “I Built a Prompt Compressor…
You Can’t Compress What You Can’t Grade
Adding an eval harness to TokenLens, and why it should have come before the compression work. A follow-up to “I Built a Prompt Compressor. The Most Useful Thing It Found Was Nothing.”
Baltimore @ National Aquarium
The short version: I built a prompt compressor, published numbers about how well it worked, and had nothing in the repo to check if those numbers were right. This is what I built to check. It immediately told me two of them were wrong, and nearly invalidated itself in the process. You don’t have to care about compression for the measurement problem to be familiar.
The last piece ended on a promise: “TokenLens has no output-quality eval harness, and building one is the next thing on the roadmap. Until then I keep the keep-rate conservative on anything that matters.”
This is that harness. It found three things. Two of them were mistakes in the article you just read. The third nearly invalidated the harness itself.
The number with no denominator
TokenLens ships with a dashboard. Until this week, that dashboard had one headline metric: tokens saved. A big, satisfying, purple number. It kept going up.
It’s also, on its own, worthless. I can trivially beat it. Set --rate 0.1 and the proxy throws away 90% of your prompt. Set it to zero and it saves everything. A metric you maximise by doing the worst possible thing is not a measurement but a temptation.
The missing number is the denominator: did the answer get worse?
This is recall with no precision. Build a tumour detector that answers “yes” for every scan and you have a model with perfect recall. It catches every case. It’s also useless, and nothing inside that number will ever tell you so. Tokens saved counts what the compressor removed and never asks what it destroyed. Retention is the other half: did the answer survive? I’d shipped a dashboard with one and not the other. It was built fifth, not first, and this is what shipping something that should have been done first but instead is done fifth teaches you.
What the harness is
The proxy has a serving loop: hot path, fails open, must never break a request. The harness is the calibration loop. It runs offline, costs real money, and answers one question: for this kind of task, how hard can I compress before the answer degrades?
task ──▶ cleartext request ──▶ model ──▶ answer A
└─▶ compressed request ──▶ model ──▶ answer B
│
answer A + answer B + reference ──▶ judge ──▶ retention
Run every task through every rung of the compression ladder at every rate and you get a quality curve. From the curve you get a policy: for each class of task, the most aggressive rate whose quality stays within tolerance of cleartext.
The metric is retention, not absolute quality. A 71/100 is alarming if the cleartext answer scored 95, and fine if cleartext also scored 71 because the task is hard. Compression is only ever accused of the difference, so the judge always sees both answers and the metric is a ratio:
retention = grade(answer from compressed input) / grade(answer from cleartext input)
Retention is capped at 1.0. Sometimes the compressed answer scores higher: pruning removed a distractor, or the model got lucky. That’s noise, and if you let it exceed 1.0, one lucky case silently pays for a real regression somewhere else in the mean. Compression is never allowed to earn credit, only to avoid losing any.
Before any model calls
The dry run (tokenlens eval --dry-run, no API calls, no key needed) shows what each rung would compress across every task.
task class arm est. tokens blocks
num-budget-01 numeric-reasoning cleartext 470 —
safe 470 0/2
llmlingua2 r=0.8 400 2/2
llmlingua2 r=0.6 323 2/2
llmlingua2 r=0.4 220 2/2
code-safety-01 code-context cleartext 188 —
safe 188 0/1
Before a single model call, the safe row jumped out. It saves zero tokens on any of the tasks. It only strips whitespace and decorative junk, and they're all tidy to begin with. Which is what it's supposed to do. But it also means the always-on, model-free safe floor, on well-formed input, does nothing. It earns its keep on messy real prompts, not on prose someone wrote carefully.
The 40% I never measured
The first article ended with a claim: “Run one of those through the bench command and the language-model codec prunes 40% or more of the tokens.” The command I printed was --rate 0.7.
That was an assumption, not a measurement. And the arithmetic was always going to contradict it: --rate is the keep fraction, so --rate 0.7 keeps 70% of tokens. Whatever the model does, the reduction can't exceed 30%. I published a 40% figure and a --rate 0.7 command in the same article and neither the repo nor I caught that they disagreed.
Here is the real curve, measured on the text of that article:
--rate tokens reduction
cleartext 1816 —
0.8 1540 15.2%
0.7 1384 23.8%
0.6 1228 32.4%
0.5 1057 41.8%
0.4 856 52.9%
40% is reachable at --rate 0.5, which means deleting half the tokens. At --rate 0.7 you get 23.8%. Nobody caught the contradiction because there was nothing in the repo whose job was to catch it.
The first article also claimed to know where the savings live: “long documents, RAG context, a big spec you paste in once.” So I benched a big spec I actually paste in: this project’s own DESIGN.md.
original: 2,563 tokens
method tokens saved reduction
safe 2,563 0 0.0%
llmlingua2 r=0.5 2,563 0 0.0%
Zero. At every rate. Because is_prose(), the safety gate that refuses to touch anything resembling code, disqualifies a block on a single code fence or two "codey" characters. A real spec is markdown: fenced blocks, tables, pipe characters everywhere. One fence throws out the whole document.
That's the gate doing exactly what it was built to do. It's also the payoff case evaporating. "Long documents" was too broad a claim. The savings live in long unstructured natural-language prose: a transcript, an article, an email thread, raw retrieved passages. Put a markdown table in it and TokenLens correctly, silently, saves you nothing.
The same lesson as the first article. The most useful thing the tool told me was where not to bother. This time, something in the repo was built to ask.
The result I nearly shipped
Then I ran the full calibration. At keep-rate 0.6 the prompts came out 32.6% smaller and the judge said quality held at 95.7%. A third off, four percent worse. That’s a shippable trade, and I was ten minutes from writing it up as one.
A boring question stopped me: how much does 95.7% move if I run it again?
I had one sample per cell. The model answers differently every time. The judge grades differently every time. I had built an instrument with no idea what its own error bar was. So I added an arm that does nothing:
@dataclass(frozen=True)
class Arm:
"""One rung of the ladder at one setting — a candidate policy.
`control` is not a rung. It compresses nothing and asks the judge to
compare two independently sampled cleartext answers to the same request.
Its retention *should* be 100%; whatever it actually is, is the noise
floor of the instrument — the model's own sampling variance plus the
judge's inconsistency. Any arm scoring inside that band has not been
measured, it has been guessed at. Without this row you cannot read the
others.
"""
The control arm sends the prompt uncompressed, samples a second answer to the same prompt, and asks the judge to grade the two against each other. Nothing was deleted. True retention is 100% by construction.
The judge gave it 93.8%.
Six point two percent of “quality loss” was free, on prompts that were never compressed. My tolerance was one percent. I had been trying to measure a 1% effect with an instrument that wobbles 6%. Every number in the run had to be re-read:
arm reduction quality what it means
control 0% 93.8% the error bar — nothing was compressed
r=0.8 14.1% 95.2% inside the noise — unresolved
r=0.6 33.9% 90.5% real loss, but only just visible
r=0.4 50.6% 60.5% real, and enormous
The result I nearly published (32.6% smaller at 95.7% quality) was one draw from a distribution I’d never looked at. Run three times and averaged, it came back as a 9.5% loss. It was never a shippable trade, just a coin landing well.
ML has a name for this: label noise. My judge is the annotator, and I’d skipped the step no one running a labelling pipeline would dream of skipping: measuring inter-annotator agreement. Hand the same pair to the same grader twice and it won’t agree with itself. Once the labels are noisy, there’s a floor under your error that no amount of model improvement gets beneath. You can’t score a difference smaller than the disagreement in the thing doing the scoring. Report a number below that floor and you’re publishing fiction with decimal places.
And the noise varies with the task:
task class control (should be 100%)
extraction, chat-history, code-context 100.0%
instruction-following 97.5%
numeric-reasoning 94.7%
long-doc-qa 88.9%
summarization 84.2%
On summarization the judge marked an uncompressed answer down 16%, and the compressed arm at keep-rate 0.8 scored 86.4%, better than the control. On that class the noise doesn’t just contaminate the signal, it exceeds it. No result on summarization means anything, at any rate, and the harness now says so instead of printing a number.
Extraction is the opposite. One right answer, consistent model output, consistent judge. The control comes back at 100.0%. On that class a quality reading is worth something. I checked.
Building a judge that can’t cheat
An LLM-as-judge has to be constructed so it can’t tell you what you want to hear. The judge is blind. It’s never told which answer came from the compressed side. Tell it and you’re measuring the judge’s prior about compression, not compression itself. There’s a test for this:
def test_judge_never_sees_which_answer_was_compressed():
...
prompt = client.judge_prompts[0].lower()
assert "compress" not in prompt
The A/B position is randomised but deterministic. Judges have position bias, so the compressed answer is shown as A or B based on a hash of the case key. Position bias can’t systematically favour one arm, and reruns shuffle identically, so the eval stays reproducible:
def _swap_for(key: str) -> bool:
return hashlib.sha256(key.encode("utf-8")).digest()[0] % 2 == 1
The judge reads the original request, not the compressed one. Show the judge the pruned prompt instead and an answer that faithfully reflects the pruned prompt looks correct: you’ve graded the compressed answer against the compressed question and confirmed they agree. Of course they agree. The point is to find out what compression threw away, so the judge grades both answers against what you actually asked, with a reference answer as ground truth.
The last guard forces the judge to reason before it scores. The response is constrained by a JSON schema, and reasoning is deliberately the first property. Schemas fill in order, so the judge articulates the comparison before it can emit a grade:
JUDGE_SCHEMA = {
"type": "object",
"properties": {
"reasoning": {"type": "string", # ← forced to argue first
"description": "Two or three sentences comparing…"},
"grade_a": {"type": "integer", "description": "Quality of ANSWER A, 0-100…"},
"grade_b": {"type": "integer", "description": "…the same standard as A."},
"note": {"type": "string", "description": "The single biggest difference."},
},
"required": ["reasoning", "grade_a", "grade_b", "note"],
"additionalProperties": False,
}
One architectural rule separate from the judge: the harness calls compress_request(), the exact function the proxy calls. It doesn't reimplement compression. If the harness and the proxy ever disagree about what gets compressed, the harness is lying to you, and a lying eval is worse than no eval, because you'll believe it.
Three things the noise floor changed
The control arm rewrote the logic for reading the results.
The tolerance bar has to move. A 99% tolerance means “keep 99% of cleartext quality”: coherent for a perfect judge, incoherent for one that scores the uncompressed answer at 93.8%. Hold the compressed arm to a standard the uncompressed arm can’t meet and you’re measuring the judge and billing the compressor. So the bar drops by the noise floor, per class, and only the loss beyond it gets charged to compression. On classes where the control reads 100%, nothing changes: you still get exactly the 99% you asked for. On numeric-reasoning the bar becomes 93.7%. Classes with steady judges get held to the strict standard, because they can be.
Quality can’t improve as you delete more tokens. My first policy picker took the arm with the biggest savings that passed. On a noisy curve it cheerfully picked keep-rate 0.6 over keep-rate 0.8 on numeric-reasoning: an arm that deleted more tokens and scored better, which is not a finding but an artifact of noise. The picker now walks from the gentlest arm upward and stops at the first failure. Anything that “passes” beyond a failure is a lucky draw, and promoting it is how a coin landing heads becomes the rate you ship.
An unmeasurable class gets refused, not certified. Apply the first fix blindly to summarization and its 15.8% noise floor drags the bar to 83%, at which point everything passes. So there’s a ceiling: a control arm that loses more than 10% is a coin, not an instrument, and the class comes back none with the reason attached. The harness exits non-zero on an unmeasurable class, so you can run it in CI and let it fail the build.
From curve to policy
Three fixes later, the list is shorter than the one I started with:
✓ extraction llmlingua2 rate=0.8 14.9% smaller at 99.8% quality
✓ code-context llmlingua2 rate=0.6 2.9% smaller at 100.0% quality
? numeric-reasoning llmlingua2 rate=0.8 14.0% smaller at 96.8% quality
provisional — inside the 5.3% noise floor
? instruction-following llmlingua2 rate=0.8 15.3% smaller at 99.0% quality
provisional — inside the 2.5% noise floor
— long-doc-qa none — unmeasurable, control lost 11%
— summarization none — unmeasurable, control lost 16%
— chat-history none — no arm held quality
One line is a real, defensible result: extraction at keep-rate 0.8, 14.9% smaller at 99.8% quality. It’s trustworthy for one reason: extraction’s control reads 100.0%. The instrument was clean when it took that reading. That’s the only sentence in this project I’d put in production.
A ? records that the harness could not detect harm, which is a claim about the instrument rather than a promise about the compressor. The report says that in those words every time, rather than printing a checkmark and letting you infer.
The default policy across everything is a provisional 14% at keep-rate 0.8. The first article shipped at keep-rate 0.7 and told you it was getting 40%. It was getting 23.8%, at a quality cost nobody had measured, because nobody had built the thing that could.
Putting quality on the dashboard
The offline harness tells you what was true when you calibrated. Traffic drifts. So the same judge now runs in shadow mode on live requests. On a sampled fraction of the ones the proxy actually compressed, it replays the original uncompressed prompt and hands both answers to the judge. The dashboard grows a Quality retained gauge next to tokens saved, with a live feed of the judge’s one-line notes. A regression now tells you what dropped, not just that something did.
A judged request is a full extra completion plus a judge call, roughly double the spend. So it’s off by default, sampled, and only fires on requests that were genuinely compressed. The proxy prints a warning at startup that says so in capital letters.
The cost is part of the point. The free alternative is where I started: a big purple number that goes up when you make the product worse.
One promise from the last article needs amending, narrowly and in public. I wrote that “prompt bodies are never logged; it records counts, not content.” With --judge on that stops being strictly true: the judge's one-line note is derived from your content ("one answer drops the deadline") and it's displayed on the dashboard. It's the only content-derived thing TokenLens ever surfaces. It's off by default and never leaves your machine, and it exists because a quality number that can't tell you what broke isn't much of a quality number. But it's an exception to a sentence I published without one.
Running it
Step 1: Prove the plumbing (free, no key)
python3 -m tokenlens eval --dry-run
Every task, what each rung would compress, how many blocks were eligible. No API calls.
Step 2: One task, one rate (~5 model calls)
export ANTHROPIC_API_KEY=sk-ant-...
python3 -m tokenlens eval --class numeric-reasoning --rates 0.6
Proves the judge returns parseable grades before you spend anything real.
Step 3: The full calibration (330 calls · $1–2 on Haiku)
python3 -m tokenlens eval --rates 0.8,0.6,0.4 --repeats 3 \
--model claude-haiku-4-5 \
--judge-model claude-haiku-4-5 \
--tolerance 0.99 --out tokenlens-eval.json
--repeats is not optional. At --repeats 1 you get a number with no error bar, which is the mistake this article is about. It costs N× as much, and the extra spend buys the error bar.
Step 4: Serve, and keep watching (2× on sampled requests)
python3 -m tokenlens serve \
--compress llmlingua2 --rate 0.6 --measure \
--judge --judge-sample 0.25 \
--eval-report tokenlens-eval.json
# in the shell running your client:
export ANTHROPIC_BASE_URL=http://127.0.0.1:8787
What it still doesn’t tell you
The judge is a model, and nobody has checked its grades against a human. Ten synthetic tasks, written by the same person who wrote the compressor. That’s the conflict of interest, right there. Retention within tolerance is a mean; the default policy’s worst case is 73%, so a mean that clears the bar can be hiding a task it wrecked. And the harness emits a per-class policy the proxy can’t yet apply per request automatically. Today you read the curve and set the rate by hand.
Three repeats is not many. Three repeats told me the noise floor is roughly 6% and unmistakably not zero, which is the finding. It can’t tell me whether keep-rate 0.8 costs 1% or 5%, and no amount of staring at my numbers will. That gap closes with more samples, a stronger judge, or both. It closes on your traffic, not mine.
All of which is fine. None of it was true of the thing I had before, which was a number that went up.
Before you read a metric, ask what it reads when nothing changed. That question is the control arm, and I shipped “tokens saved” for weeks without asking it. The answer was six percent of noise that looked exactly like signal, and every number I’d published had to be re-read once I knew.
The compressor came out of this untouched. I came out with two corrections to a published article and one shippable trade that turned out to be a coin flip. That’s what you get for building the eval fifth.
By Joshua McDonald on July 15, 2026.
Exported from Medium on August 26, 2026.
Reader discussion