Microsoft's TokenEconomizer reports 18.4 to 30.9% lower mean cost on SWE-Bench, cheaper in all 30 repository × model cells, and Pass@1 up in 4 of 5 combinations. I ported the idea to DeepSeek Harness, ran a first A/B test, and got nothing I could defend. That result stays in the record: twelve sessions, plugin on versus off, +0.5% at the group level, the two short tasks pointing in opposite directions, all of it inside a pre-registered ±15% noise band.

The honest reading was that my experiment could not see the effect even if it was there. So I rebuilt the measurement, pre-registered a verdict rule, and ran it again: 24 matched pairs, 48 sessions, 139 paired steps, one frozen starting state. This time the plugin saves tokens, and I can say how much.

What was wrong with the first test

Three things went wrong, and only one was sample size.

One-shot read-only sessions. Every first-test run was a single-turn subagent session that read files and stopped. Tool pruning never fired because the agent never got far enough, so the two most interesting mechanisms had nothing to act on. I was measuring the thinnest mechanism, a per-session micro-skills injection, in the setting where it matters least.

Cache noise at n = 2. The primary metric was inputTokens + outputTokens, and uncached input mostly measures cache behavior: B1 posted 3.3× B2's uncached input on the same task, while total prompt volume moved 6%. With two repeats per cell, that noise was the result.

No verdict rule before the data. After the fact, everyone can find a reading. I wanted the decision made before the runs: what counts as "saves", what counts as "doesn't", and what counts as "inconclusive".

What I changed

Step 1: pre-heat, then freeze. Tool exposure is driven by usage history, so every session starts from a warm state rather than an empty one. I ran six setup sessions to populate the plugin's state, then waited 62 minutes (3,728.8 seconds of measured idle time) so the tool-pruning pass and the fact distillation had completed, then snapshotted the state. Every A and B session after that started from that snapshot. The plugin state was never written to during the experiment.

Step 2: 24 matched pairs, alternating order. Six tasks, four repeats each, two arms. The arm order alternates AB, BA, AB, BA within each task. Same model (opencode-go / deepseek-v4.1-flash, reasoningEffort=max), same repo, same task prompt, same headless tool surface. A is the plugin on; B is the plugin with enabled: false, everything else identical.

Step 3: the verdict rule, written down first. The test counts as a save only if both hold: the 95% bootstrap CI upper bound on the per-step prompt-token delta is below zero, and the absolute saving is at least 300 tokens per step. This is what stops a 0.3% result from being reported as a win.

I also checked that the mechanism actually fired. A token difference with no mechanism behind it is a bug. The frozen A-arm header carried 15 tools and 16,503 bytes, with 8 of the 23 tools denied by pruning; B carried 23 tools and 19,997 bytes. A also injected 214 characters of micro-skills context into every session from the second step on. No injection reached B.

What the 24 pairs said

The primary metric is prompt tokens per step: inputTokens + cacheReadTokens, counted per model call, so a long session and a short session contribute the same unit. Across 139 paired steps:

A spends 93.37% of what B spends per step. The median paired difference is −804 tokens per step, and the mean of the 24 pair-level means is −842.5, so the result isn't one or two dramatic pairs dragging the mean.

Two sub-results:

• Step 1: −904 tokens, or −8.05%. The first model call of a session has no conversation context yet, so this isolates the tool-schema difference almost perfectly. It is also deterministic: every pair shows the same −904, which is why the interval collapses.
• Step 2 onward: −842.9 tokens, or −6.38% (95% CI [−8.17%, −4.37%]). On later steps conversation content is present, and the relative advantage shrinks slightly but stays below zero.

Every one of the six tasks moved the same way:

The weakest task (−4.68%) and the strongest (−9.34%) differ by roughly the size of the effect itself. That is a warning about per-task claims: six tasks, four repeats, and one repo cannot support "the plugin saves 9% on refactors."

Total tokens per session tell a noisier story: 2,053,523 in B against 1,902,222 in A, a −7.37% reduction, but the paired 95% CI is [−15,761.5, +3,215.3] and crosses zero. Total tokens include output length, which the plugin does not control, and cache behavior, which varies with conversation shape. That is why the per-step prompt metric was pre-registered as primary.

Quality held: 48 of 48 sessions passed verification, 24 of 24 pairs were usable, and no session was excluded. Output tokens were essentially identical, A 39,827 in total against B 40,315, so nothing here is paid for by writing less.

What this does not prove

• One model, one repository, six short tasks, headless. The headless tool surface is 23 tools in B. A desktop install loads 59, so the absolute bytes removed per request would be larger, but I did not run that configuration and I am not going to multiply numbers I did not measure.
• Two mechanisms, one switch. The A-arm advantage combines tool-schema pruning, fixed per request, and micro-skills injection, 214 characters once per session. This experiment treats "plugin on/off" as one unit and does not separate the two.
• A pooled ratio estimator with a cluster bootstrap. The headline −6.63% is ΣΔ / ΣB pooled across 139 steps, with a 95% CI from 10,000 bootstrap resamples clustered by the 24 pairs. Per-step deltas are correlated within a session, so the cluster is the pair rather than the step.
• Short sessions. The whole experiment is 139 model calls. It says nothing about multi-hour sessions, where per-request schema savings compound while context growth dominates.

The film, the GIFs, the code

I turned this second round into a 90-second film with HyperFrames, HTML and CSS animated to a timeline, on-screen text only, no voiceover, one music bed. It opens with the number, then covers the failed first test, the redesign, and the result.

https://youtu.be/F82yTG_Vpaw

Two clips from the film, made with gif-master's video-to-GIF converter: the three-step design and the result panel.

The plugin is public: github.com/Mng12345/dsh-tokeneconomizer. It is Apache-2.0, the README documents the DSH hook mapping and configuration, and lib/ is committed so you can install it without a build step. The benchmark, raw session logs and preregistration are in a private repository; the design, the verdict rule and every number above come from it.

Last thing

The first post ended with "the cost of just being an agent dominates anything a thin layer can shave off." That was too strong, and it described my measurement more than the plugin. With a warm frozen state, matched pairs and a rule written before the data, the thin layer is visible: 853 tokens per step, 6.63%, with the interval and the mechanism both behind it.

The number is small. It is also about the size you would expect from deleting eight tool schemas from every request.