> blog // 12 November 2026
Every test passed. The fix was still wrong.
AI builders fixed planted bugs in our made-up code. Our checker passed 21 of 21 fixes, and 15 still got things wrong that no test asked about. With a written plan first, 34 of 34 runs failed none of those hidden cases. What that means if you pay for software.
When AI writes the code and the tests pass, you know one thing: the code does what the tests ask. You don’t know what the tests forgot. In a trial on our own made-up code, AI builders’ fixes passed their tests and our AI checker 21 times out of 21 (3 more tasks never built because of model errors, and weren’t rerun). Fifteen of those fixes still got something wrong that no test had asked about. When another AI wrote a plan before the builder started, 34 runs out of 34 failed none of the hidden cases. The cheapest plan that did it cost about two US cents.
Testing on planted bugs is a weaker claim than it sounds. We wrote the bugs, so we knew the right answer and wrote the extra checks to catch the wrong ones; real bugs come with no answer key. The code was a small made-up library, not a client’s. Each mix of planner, builder and bug ran only once or twice with a plan, and four times without, so these counts show a direction, not a rate to quote. The counts come from the report of the AI agent that ran the trial; the agent that oversees our engineering read it but didn’t rerun the runs. And AI agents did the building: the builders, the checker, the planners and the agent that ran the trial are all AI models, and AI agents wrote and built this page.
A fifty-cent invoice printed “$0.5”, and every check said pass
We first saw it earlier that morning, on a different made-up bug. Issue 5 asked for a minus sign to be fixed in the money code. Qwen 3.8 27B, one of five AI builder models we tried on it, fixed exactly that. It met the issue’s own test of done, and the checker passed it. A fifty-cent invoice still printed “$0.5”. The other four also fixed the $0.5.
issue 5
So we went looking for it on purpose.
Our checker passed all 15 fixes that were still wrong
We planted three more bugs, each one where the obvious fix is too narrow. Two AI builder models, GLM 5.3 Flash and Qwen 3.8 27B, fixed each one. The other model then checked the work by running the tests again. Around each bug we wrote hidden cases: neighbouring cases the issue didn’t name, which the builder and the checker never saw.
Each square is one builder run. Flip the switches.
What our checker saw: 21 of 21 runs passedWhat the hidden cases saw: 15 of those 21 were still wrongWith a plan, what our checker saw: 33 of 34 runs passedWith a plan, hidden cases failed in 0 of 34 runs
The checker isn’t broken. It reruns the tests, and the tests passed. A fix that is too narrow passes every test anyone wrote, because nobody wrote a test for what it missed.
The second switch shows runs where, before the builder started, another AI model read the code without changing it and wrote a plan: which files to change, what else depends on them, what to check and what not to touch. In these 34 planned runs, no hidden case failed. With no plan, 6 of 21 runs were clean. The two sets aren’t matched cohorts, so this shows a direction on three bugs; it doesn’t prove that a plan fixes it.
The checker earned its keep once. In one run with a plan, the builder deleted an existing test and wrote new ones in its place. The checker failed it for removing a test; our hidden cases scored it clean. The two checks see different faults, so we keep both.
The counts behind the grid, as a table:
| Runs | Issue 6 | Issue 7 | Issue 8 | All three |
|---|---|---|---|---|
| No plan: built | 7 | 7 | 7 | 21 |
| No plan: checker passed | 7 | 7 | 7 | 21 |
| No plan: failed hidden cases | 5 | 7 | 3 | 15 |
| With a plan: built | 11 | 12 | 11 | 34 |
| With a plan: checker passed | 10 | 12 | 11 | 33 |
| With a plan: failed hidden cases | 0 | 0 | 0 | 0 |
Issue 7: every builder patched one file, and the fix needed four
Issue 7 is the clearest case. The price of one unit is rounded in pricing.ts, and three other files use that rounded price. With no plan, 7 of 7 built runs changed order.ts, the file the issue named, and nothing else. Each failed 6 hidden cases. With a plan, 12 of 12 changed all four files and failed none.
No plan: 7 of 7 runs
With a plan: 12 of 12 runs
The plans put the cause one step upstream, in pricing.ts. The helper agent read the openings of 16 of the 20 plans, and each of those named the cause; it checked the other four only by the files they name. None of the 11 plans from the first two planners proposed a narrow fix.
The cheapest plan that worked cost about two US cents
We tried three AI planners. On these bugs every one of them led to clean fixes, so a dearer plan bought nothing we could measure here. Whether a strong planner beats a cheap one on harder, real work is the next question; these bugs were too small to show it.
We kept the checker and put a plan in front of the builder
A pass used to end the job
- Builderwrites the fix
- Checkerreruns the visible tests
- Passread as done
A plan now names the work first
- Plannerreads the code, names the files and what depends on them
- Builderworks from the plan
- Testswritten from the plan's list of what depends on the changenext: designed, not yet measured
- Checkerstill reruns the tests; a pass is not proof the fix is complete
A checker’s pass now means the tests the builder could see still pass. It no longer means the fix is complete.
Ask whoever builds your software who checks the checker
Ask these of a person or an agent, whether you’re paying for software or making it with AI:
- Show me the plan before the code. Which files change, and what else uses them? A cheap plan cost us 1 to 2 cents; the strong planner we now default to cost 32 to 68 cents.
- What did you test that I didn’t ask for? The fifty-cent invoice was beside the bug, not in it.
- Who wrote the tests? If the builder wrote them, they share its blind spots.
- What would a pass miss? Our checker missed the narrow fixes and caught a deleted test. Each check sees something the other can’t.
Three planted bugs are a direction, not proof on client work
- The code: a small made-up TypeScript library with three planted bugs, run on 11 November 2026 in a test office on Cloudflare. The builders were GLM 5.3 Flash and Qwen 3.8 27B, each checked by the other. Each built change was then run against the hidden cases, outside the office.
- The sample: one or two runs per mix of planner, builder and bug with a plan, four without. It shows a direction on three bugs we planted. It doesn’t rank the models or the planners.
- The counts are from the report of the AI agent that ran the trial. The agent that oversees our engineering read that report and didn’t rerun these runs.
- The costs are list prices times the tokens each model reported, not a bill.
- The invoice comes from a separate run that morning: issue 5, five builder models.
- The raw runs aren’t published. Ask us for the run sheet.
Jezweb builds websites and custom apps for Australian businesses. If you’re paying for software, or making it with AI, and want to talk about how it gets checked, get in touch.