What Still Saved Tokens After the Rebuild, and Which Experiment I Stopped

After the rebuild, the implementation agent in my Flutter project, the developer from Part 1, preloaded 3,034 lines of adapter before it opened its first file: coding guidelines, test strategy and gate rules. My wish was an obvious one. The guidelines should be split by domain, meaning backend SDK, state management, styling and general, and the agent should load only what a step needs.

Looking Back at Part 1 and Part 2

In Part 1 I described two dividing lines: which knowledge belongs in core, adapter or content, and by which route it reaches which agent. In Part 2 an audit showed that my own setup loaded all twelve rules into every agent. Since version 2.0, eleven of them have been skills that only the roles needing them preload, and the base load of the main session dropped by 84% in the template. In the first real project the implementation agent started only slightly lighter, because its adapters are large.

The Experiment That Said "No"

An earlier experience made me cautious. A text rule telling agents to use the code index first (CodeGraph in my case), meaning the queryable symbol and call graph of the project, had led to zero index queries in the first run. "Load the skill before the first change" would be another text rule of that kind. So I measured before rolling it out. A stop threshold that is fixed in advance turns a wish into an experiment.

The undertaking had three stages: first repair the measurement and freeze a measurement plan, then trim without losing anything, and only after that load on demand, but only if an experiment met the go criteria.

For the experiment I needed a tool that reads transcripts. A small Python script reports, per subagent run, the agent, the turns, the token total and the order in which skills were loaded. For each domain it shows whether the matching skill was loaded before the first change in that domain. I call this share the load rate.

What a Run Costs Before It Starts

Before planning the experiment, I analysed 184 old subagent runs from the Flutter project, including 21 by the implementation agent and 30 by the code reviewer. A run in the experiment consists of implementation and code review. In the old runs that cost about 182 million tokens at the median: 171 million for the implementation, which needed 319 turns at the median, and 11 million for the review. 97 to 99% of that was cache reads, meaning context that comes from the prompt cache and is billed considerably cheaper than freshly read tokens.

The startup load sits in the context again in every turn. Multiplying the startup context by the number of turns, it made up, at the median, about 36% of the tokens of an implementation run and about 60% of a review run. Every token saved on startup load therefore counts in every turn.

The full design would have needed three tasks in two variants with three repetitions each, so 18 runs and, extrapolated, about 3.3 billion tokens. The spread of the old runs was above 30%. Three repetitions would not have made a 10% difference visible anyway. I settled on a pilot with six runs and planned to decide again afterwards.

The Setup

A subagent searched the Flutter project for three pieces of work that were already implemented and had an approved plan and a start commit: a piece of data logic (three source files, four test files), a UI component (three source files plus catalogue, six test files) and a change to existing code (two source files, five test files).

Along the way a confounding factor turned up. Today's guidelines already contained the model solutions, because they had been written back into them afterwards. Both variants therefore got adapters as of the start commit. Variant A preloads them as before. Variant B has the same content, split into domain skills that the agent is supposed to load itself. I checked that B has identical content with a diff of the sorted lines.

The go rule was fixed in advance, and all three conditions had to hold:

• Load rate at least 90%.
• Quality no worse: at most one additional major finding or one additional fix round in total.
• Tokens on average at most 90% of variant A.

The experiment stops after the pilot if the load rate is below two thirds or if variant B is not cheaper in two of three tasks.

The runs took place in throwaway clones, not in worktrees. A worktree shares the git config, the stash stack and origin with the real repository, and the implementation agent pushes on its own. Each run looked like this:

git clone <target-project> runs/R1a-A && cd runs/R1a-A
git remote remove origin          # push fails loudly, no CI starts
# copy .claude/ and scripts/ from the template branch, insert adapter set
git config core.hooksPath scripts/git-hooks
claude                            # once, interactively: confirm the trust dialog
claude -p "$(cat prompt.txt)" --output-format json > run.json

Every new folder needs the trust dialog once. Without it, Claude Code ignores the permissions.allow entries in the project settings, and it tells you so. During one run the machine went to sleep. The subagents ran to completion, but the controlling session stayed incomplete. For long headless runs on macOS, caffeinate -i therefore goes in front of the command.

The Result

The load rate was 0 of 4. The denominator is the pairs of run and touched domain in variant B: three runs, one of which made changes in two domains. In no case did the agent load the domain skill before the first change. It only opened the domain skills at the end, to write a rule into them.

Variant B started 11,000 to 15,000 tokens smaller but needed more turns. Total tokens were 7, 9 and 22% above variant A, with one run per task. Quality was the same in both variants: no major finding, no fix round.

The projection of 182 million tokens per run did not hold. The real runs cost 2.6 to 14.5 million tokens, all six together about $35. The reference tasks were small, while the old runs had been real features.

The stop rule was met, so the answer was no-go. Everything stays preloaded. Why I would rather entrust obligations to a mechanism than to a sentence in the prompt is covered in the post on prohibitions as a mechanism.

What Helped Instead

The second stage of the undertaking was lossless trimming. It was running anyway, and it worked:

• The instructions for agent memory shrank from 208 to 60 lines. Claude Code injects them itself when memory: is set. The subagent docs say: "The subagent's system prompt includes instructions for reading and writing to the memory directory."
• A copy of the PR instructions in the git skill and a duplicate table of status transitions were removed.
• The lines "First: invoke skill X" can go when the skill is preloaded anyway. These are the computed duplicates from Part 2.

In the template, this took a pipeline run from 165,777 tokens in version 2.0 to 121,517 in version 2.1.

On top of that came size warnings in the adapter check script, which I introduce in the post on the adapter contract. It warns when project-context.md has more than 200 lines or a preloaded adapter skill more than 500. The two limits follow the recommendations in the docs: under 200 lines per CLAUDE.md (Memory) and under 500 lines for the main body of a skill (Skill best practices). It remains a warning and does not set an exit code.

A behaviour check with two more runs, $4.30 in total, showed that the trimming broke nothing. Gates, status changes, review and push went through, and no agent loaded a preloaded skill a second time. It also showed that the implementation agent never invokes two procedure skills on its own: the one for the PR handover and the one for status changes on the documents. Since version 2.2 it preloads both, which costs 7,880 tokens per run in the template, 6.5% more. I deliberately traded tokens for reliability here.

The other way round, a skill can cost nothing at all. My skill for the best-practice audit carries disable-model-invocation: true, and according to the skill docs not even its description is in the context then. It only loads when I invoke it.

The Numbers

The jump came with 2.0. After that, the size of the adapters decides. The tables show what I base that on.

In the Template

I measured all five releases with the same revision of measure-context.py. With empty adapters, the script counts only the configuration the workflow itself brings along, estimates tokens as characters ÷ 4 and assumes fixed call counts for a run. The intermediate states are in the text above: 2.1 lowered the startup load per run to 121,517 tokens, 2.2 raised it to 129,397 with the two preloaded procedure skills. From 1.5.2 to 2.3.0 the base load dropped by 83% (140 tokens were added after 2.0) and the startup load per run by 58%.

In Two Real Projects

In the Flutter project (target project A) and in a Nuxt project (target project B), I built one feature each on version 2.3.0 and compared the startup load with the old runs. What I measured is the context in the first turn: the sum of input, cache creation and cache reads of the model's first response in the transcript, so the whole startup context as the API reports it. The values are medians, and n is the number of runs.

From 1.x to 2.3.0 the startup load of the main session dropped by 68% (A) and 45% (B), that of the use case reviewer by 72% and 59%. For the implementation agent it was only 21% and 8%, for the code reviewer 29% and 21%. The runs under 2.0 to 2.2 already sit at the level of 2.3.0 in both projects, in target project A for example the main session at 65k and the implementation at 156k. So versions 2.1 to 2.3 barely moved the startup load. The implementation agent and the code reviewer stay heavy because they preload the large adapter skills. In target project A, coding-guidelines had 1,674 lines at the time of the feature runs and testing-strategy 776, in target project B 184 and 411. Accordingly, the implementation in B starts with 132k instead of 159k. Target project B already had smaller adapters under 1.x and therefore started lower, so the savings there are smaller.

Template and project measure different things. The template counts only the configuration with empty adapters. In the project, everything the API reports as context in the first turn counts, including the system prompt, tools, filled adapters and memory. In the template the base load dropped by about 19,000 tokens (−83%), in the Flutter project by 137,000 (−68%). The absolute gain is larger in the project because under 1.x the large adapter files also sat in every session as rules.

What a Feature Costs Today

Total tokens is the sum of usage across all responses in the run, main session and subagents together. The two features differ in size, so the numbers cannot be read against each other, nor against the old runs. In target project B, 26.9 million go to mirroring the docs to Confluence, which the post on repo, hub and mirror covers.

Has It Gotten Worse?

Both feature runs passed code review without a critical or major finding, and the gates ran green. For the small feature in A they ran in full after two of the three plan steps, for the large one in B in full after steps 5, 10 and 17, with subsets in between. In the experiment, too, quality was the same in both variants.

This evidence is only of limited reliability. It is one run per project, and a controlled comparison against 1.x with the same feature is missing. The startup load is comparable, total tokens and runtime are not.

What Is Still Open

The largest remaining item of startup load in the Flutter project is the preloaded adapter. The check script warns there for project-context.md with 389 lines, for coding-guidelines, for testing-strategy and for styling-guidelines with 1,236 lines. In the Nuxt project project-context.md has 277 lines against a budget of 200. I have not trimmed them yet. The template for coding-guidelines also still contains a section with canonical patterns, that is, positive directives.

What I Would Recommend to Others

• Measure first: base load, startup load per agent and duplicates via script, then cross-check the estimate against real transcripts.
• Load as a rule only what every agent needs, and check the loading behaviour of each mechanism against the current docs.
• Preload role knowledge via skills: and make sure the preloaded adapters stay small.
• Before changing behaviour, run a small experiment with a stop threshold set in advance. "Doing nothing" is a valid result.

How do you measure what your agent setup costs in context before the first line of code is written?