Compressing by 20 % is free. Compressing by 50 % is not.
300 calls, 10 documents in French and English, five compression rates, a judge model and a real Azure deployment. What a token classifier saves, and where it starts to cost.
We ran the same question against the same document five times: raw, then with 15 %, 31 %, 46 % and 62 % of the context removed by a token classifier (LLMLingua-2, mBERT). Ten documents — meeting notes, SaaS analyses, long reports, in French and English, plus three MeetingBank transcripts — three questions each, two repetitions. 300 answer calls to an Azure OpenAI deployment, and as many judge calls scoring each answer 1, 0.5 or 0 against a reference.
Quality holds at −20 %, bends at −50 %, breaks at −65 %
On the seven non-MeetingBank documents the mean score goes from 0.98 raw to 0.95 with 15 % removed, 0.94 at 31 %, 0.89 at 46 % and 0.69 at 62 %. Removing a fifth of the context costs three points and produces zero wrong answers. Removing two thirds turns one answer in six into an error.
| Rate kept | Tokens sent | Mean score | Correct | Wrong |
|---|---|---|---|---|
| 1.0 (raw) | 5,954 | 0.91 | 87 % | 5 % |
| 0.8 | 5,052 (−15 %) | 0.93 | 88 % | 3 % |
| 0.65 | 4,092 (−31 %) | 0.88 | 83 % | 8 % |
| 0.5 | 3,225 (−46 %) | 0.88 | 82 % | 7 % |
| 0.33 | 2,249 (−62 %) | 0.76 | 65 % | 13 % |
Two things decide where the cliff is. Factual questions — numbers, names, lists — hold to 46 % removed because digits are protected by the classifier. Nuance questions — a condition, a cause, a reason — start dropping at 15 % (0.93 → 0.86) and collapse at 62 % (0.54). And French degrades before English: on nuance questions English stays at 1.00 down to rate 0.5 while French is already at 0.80 at rate 0.8. The classifier is supervised on English; that shows.
What it saves on the invoice
On a cold call — the first time a prefix is sent — compression delivers what it promises: −25 % at rate 0.8, −30 % at 0.65, −51 % at 0.5. On a warm call, when the provider's prompt cache already holds the document, the saving disappears: +2 % at 0.8, +16 % at 0.65, −3 % at 0.5, +38 % at 0.33.
The reason is the output. The model answers longer when the context is shredded (+23 % output tokens at rate 0.8, +139 % at 0.33) and starts reasoning to decode it: zero reasoning tokens raw, 85 on average at 0.33, on 42 % of calls. Output costs eight times the input price and already weighs 45–77 % of a warm bill. This is the compression paradox of the literature, measured on a real deployment.
Compression is deterministic, so it is cache-safe: 97–99 % of input tokens were read from cache for the compressed versions too. It does not break the cache. It just cannot beat it.
No latency gain
Median time to first token was 0.86 s raw and identical at rates 0.8 and 0.5; +34–40 % at 0.33 because of the reasoning before the first token. The compressor itself costs 80 ms for 500 tokens and 2–7 s for 8–18k tokens on a laptop GPU. To win time, the prefill you skip has to exceed the compression step, and on this model the prefill of 18k tokens fits in about a second.
The rule we ship
- Default aggressiveness removes 15–20 %. It is quality-neutral on factual work and saves 12–25 % on uncached calls.
- Never recompress a prefix the provider already caches. Compress once, before the first call, and keep it frozen.
- Protect numbers, code, JSON, URLs and quoted spans. Always.
- Treat French and other non-English inputs conservatively until the multilingual model lands.
- Measure on the invoice, not on token counts. Longer outputs are part of the bill.
Limits of this run: n = 6 per document and rate; an LLM judge scoring LLM-generated references (±0.05 noise); list prices as placeholders; a single, small compressor; latency measured from Europe to a GlobalStandard deployment.