Part 1 covered the first spec of whatif-mcp, built by AI agents using sdd-kit, my spec-driven development process for Claude Code. Part 2 covers the second spec, with a closer look at token usage, reviewer findings, and what escaped review.
Two reviewer agents consumed just 9% of the input tokens. They caught implementation issues, but QA exposed a different weakness: use cases the specs never addressed.
TL;DR — If you’re building with AI agents
- Use independent reviewers. Two reviewer agents consumed 9% of input tokens and caught implementation issues, though one bug still reached QA.
- Review the spec before the code. Of 18 QA findings across two specs, 15 were gaps in the specifications. Implementation review can’t reliably catch requirements that were never written.
- Turn lessons into tests. An automated hostile-input test held up under QA. A written rule to check calculations didn’t prevent a rounding bug from recurring.
The setup
Spec 0002 adds a what_if tool. Give it a change such as “cut my salary to 3,600 a month,” and it compares the current financial projection with the modified one.
The development process follows a consistent sequence. An implementer agent builds each task test-first. A gate script runs the tests and checks. Two independent reviewer agents then examine the diff: one against the spec, the other against coding standards. I decide what gets merged.
Once the spec is implemented, an adversarial QA agent probes the completed work for issues the earlier stages missed.
Here’s how the first two specs compare:
| Measure | Spec 0001 | Spec 0002 |
| Tasks | 10 | 4 |
| Reviewer findings | 176 | 72 |
| Findings by me | 0 | 0 |
| Bugs found after merge by QA | 2 | 1 |
| Spec gaps found by QA | 8 | 7 |
Across both specs, most QA findings involved behavior the specifications hadn’t addressed.
What review costs
The four tasks in Spec 0002 consumed 15.7 million input tokens, all using Claude Opus 5.5.
| Work | Input tokens | Share | Result |
| Implementer, including rework | 14.2M | 91% | Code and tests, across 154 calls |
| First review round | 0.69M | 4% | 50 findings, 23 sent back for rework |
| Second review round | 0.78M | 5% | 22 findings, none sent back for rework |
Each reviewer typically made three to seven model calls. The implementer made 23 to 52 calls per task, with context growing as the work progressed.
The first review round consumed just 4% of input tokens and resulted in 23 findings being sent back for rework. That’s a relatively small share of the total token usage for an additional layer of scrutiny.
The second round is less clear-cut. It consumed slightly more input than the first and produced 22 findings, but none required further rework. No code changed as a result.
Could that second round be reduced or eliminated? Possibly. But it accounts for only 5% of input tokens, and this experiment doesn’t establish what would happen without it.
What review misses
The reviewer agents evaluate implementation against the specification and coding standards. They can identify code that violates a stated requirement, but they’re less likely to catch behavior the spec never considered.
The QA results illustrate the distinction. Across two specs, QA reported 18 findings: 3 implementation bugs and 15 specification gaps.
One gap involved a sequence of operations:
- Rename the Salary flow.
- Remove it using its new name.
- Create a new flow named Salary.
The result contained two entries named Salary.
The implementation followed the spec’s matching rule. The problem was that the rule didn’t account for this sequence. The implementer had followed the written requirements, and the reviewers had no explicit requirement against which to flag the behavior.
This is a different failure mode from a coding bug. More implementation review won’t necessarily find it. The missing behavior needs to be identified before the code is written.
To address that, sdd-kit 0.4.0 adds an optional adversarial spec probe. Before a spec is approved, a QA agent examines it for missing cases, ambiguities, and assumptions.
Part 3 will examine whether that additional step reduces the gaps found after implementation.
Lessons from the experiment
Three lessons stand out from these two specs.
1. Add independent reviewer agents.
The two reviewers consumed 9% of input tokens in Spec 0002. The first round alone identified 50 findings, with 23 sent back for rework.
The reviewers didn’t catch everything. QA still found one implementation bug. But the token overhead was modest relative to implementation.
2. Put the next effort into the spec.
Of the 18 QA findings across both specs, 15 concerned behavior the specifications hadn’t addressed.
Reviewing code against an incomplete spec can confirm that the code meets the written requirements while missing what those requirements should have included.
An adversarial review of the spec, before implementation begins, is the next experiment.
3. Turn retrospective lessons into tests, not just rules.
Two lessons from Spec 0001 carried into Spec 0002, with different outcomes.
A hostile-input test added after Spec 0001 held up. QA sent 1,500 random inputs to Spec 0002 without finding a crash.
A written rule to hand-check numerical results was less effective. A rounding bug appeared again.
The distinction matters: a lesson encoded in an executable test is enforced automatically. A lesson written as an instruction still depends on an agent remembering and applying it correctly.
How I measured
Claude Code reports a token count when an agent finishes. For these four tasks, those reported counts totaled 967,485 tokens.
But those figures reflect the context size at the end of each agent run, not the total number of tokens processed.
Each model call sends its context again. To measure total input usage, I summed the usage records for every model call in the agent transcripts.
That produced the 15.7 million input-token figure.
There are important limits to this measurement:
- Input tokens only. The transcripts don’t reliably record output usage, so output tokens are excluded.
- Caching matters. Cache reads accounted for 92% of input tokens. Because cached input is billed at a fraction of the fresh-input rate, token share should not be interpreted as cost share. I haven’t calculated a dollar figure.
- QA is excluded. The separate QA pass consumed another 2.6 million input tokens, outside the implementation and review totals above.
These numbers describe input-token consumption, not the full financial cost of the process.
What I’d change next
The next experiment isn’t more code review. It’s moving adversarial review earlier, before implementation begins.
If the QA agent can identify missing use cases in the spec, those cases can become requirements and tests rather than post-merge findings.
Spec 0003 will test that hypothesis.
Caveats
This is one project, built by the person who designed the process. The token measurements cover one spec with four implementation tasks.
The findings are useful for understanding where effort went and which issues escaped review, but they aren’t a controlled comparison of agent workflows. In particular, the results don’t establish how many defects would have occurred without reviewers.
The process is open source under the MIT license:
- sdd-kit — the spec-driven development process, packaged as a Claude Code plugin.
- whatif-mcp — the project built with it, including specs, review records, QA reports, and retrospectives.
Part 3: Spec 0003, solve-for, and whether reviewing the specification before implementation changes the results.
Leave a Reply