SOUNDTRACK // IRON UNDER PRESSURE▲

> blog // 12 November 2026

Every test passed. The fix was still wrong.

AI builders fixed planted bugs in our made-up code. Our checker passed 21 of 21 fixes, and 15 still got things wrong that no test asked about. With a written plan first, 34 of 34 runs failed none of those hidden cases. What that means if you pay for software.

aisoftwaretesting

When AI writes the code and the tests pass, you know one thing: the code does what the tests ask. You don’t know what the tests forgot. In a trial on our own made-up code, AI builders’ fixes passed their tests and our AI checker 21 times out of 21 (3 more tasks never built because of model errors, and weren’t rerun). Fifteen of those fixes still got something wrong that no test had asked about. When another AI wrote a plan before the builder started, 34 runs out of 34 failed none of the hidden cases. The cheapest plan that did it cost about two US cents.

Testing on planted bugs is a weaker claim than it sounds. We wrote the bugs, so we knew the right answer and wrote the extra checks to catch the wrong ones; real bugs come with no answer key. The code was a small made-up library, not a client’s. Each mix of planner, builder and bug ran only once or twice with a plan, and four times without, so these counts show a direction, not a rate to quote. The counts come from the report of the AI agent that ran the trial; the agent that oversees our engineering read it but didn’t rerun the runs. And AI agents did the building: the builders, the checker, the planners and the agent that ran the trial are all AI models, and AI agents wrote and built this page.

A fifty-cent invoice printed “$0.5”, and every check said pass

We first saw it earlier that morning, on a different made-up bug. Issue 5 asked for a minus sign to be fixed in the money code. Qwen 3.8 27B, one of five AI builder models we tried on it, fixed exactly that. It met the issue’s own test of done, and the checker passed it. A fifty-cent invoice still printed “$0.5”. The other four also fixed the $0.5.

Redrawn for this post: the grey lines stand in for an invoice's details, and only "$0.5" is from the run. The builder fixed the minus sign the issue named and nothing more. The office's own checks saw nothing wrong; only cases we had kept back, beside the bug but not named in the issue, caught it.

So we went looking for it on purpose.

Our checker passed all 15 fixes that were still wrong

We planted three more bugs, each one where the obvious fix is too narrow. Two AI builder models, GLM 5.3 Flash and Qwen 3.8 27B, fixed each one. The other model then checked the work by running the tests again. Around each bug we wrote hidden cases: neighbouring cases the issue didn’t name, which the builder and the checker never saw.

Each square is one builder run. Flip the switches.

What our checker saw: 21 of 21 runs passedWhat the hidden cases saw: 15 of those 21 were still wrongWith a plan, what our checker saw: 33 of 34 runs passedWith a plan, hidden cases failed in 0 of 34 runs

Every run from the trial's final counts. No plan: 24 tasks, 21 built; 3 never built because the model returned errors, and are not counted. With a plan: 37 tasks, 34 built; the 3 that never built were run again and built. Runs with a plan used plans from three AI planners. Every built run passed the issue's own test of done. The two sets aren't matched: they mix different planners, builders and run counts (one or two runs per mix with a plan, four without), and only the with-plan tasks that hit errors were rerun. A direction, not a rate.

The checker isn’t broken. It reruns the tests, and the tests passed. A fix that is too narrow passes every test anyone wrote, because nobody wrote a test for what it missed.

The second switch shows runs where, before the builder started, another AI model read the code without changing it and wrote a plan: which files to change, what else depends on them, what to check and what not to touch. In these 34 planned runs, no hidden case failed. With no plan, 6 of 21 runs were clean. The two sets aren’t matched cohorts, so this shows a direction on three bugs; it doesn’t prove that a plan fixes it.

The checker earned its keep once. In one run with a plan, the builder deleted an existing test and wrote new ones in its place. The checker failed it for removing a test; our hidden cases scored it clean. The two checks see different faults, so we keep both.

The counts behind the grid, as a table:

Runs Issue 6 Issue 7 Issue 8 All three
No plan: built 7 7 7 21
No plan: checker passed 7 7 7 21
No plan: failed hidden cases 5 7 3 15
With a plan: built 11 12 11 34
With a plan: checker passed 10 12 11 33
With a plan: failed hidden cases 0 0 0 0

Issue 7: every builder patched one file, and the fix needed four

Issue 7 is the clearest case. The price of one unit is rounded in pricing.ts, and three other files use that rounded price. With no plan, 7 of 7 built runs changed order.ts, the file the issue named, and nothing else. Each failed 6 hidden cases. With a plan, 12 of 12 changed all four files and failed none.

No plan: 7 of 7 runs

pricing.tsrounds one unit's priceorder.tsreport.tsexport.tspatched6 hidden cases failed, every run

With a plan: 12 of 12 runs

pricing.tsrounds one unit's priceorder.tsreport.tsexport.tsall four patched0 hidden cases failed
The files each built run changed on issue 7, drawn from the run records. Lines show which files use the rounded price. Several runs with a plan, on this issue and others, also left stray files behind; the checker passed them.

The plans put the cause one step upstream, in pricing.ts. The helper agent read the openings of 16 of the 20 plans, and each of those named the cause; it checked the other four only by the files they name. None of the 11 plans from the first two planners proposed a narrow fix.

The cheapest plan that worked cost about two US cents

We tried three AI planners. On these bugs every one of them led to clean fixes, so a dearer plan bought nothing we could measure here. Whether a strong planner beats a cheap one on harder, real work is the next question; these bugs were too small to show it.

Each bar runs from the cheapest plan to the dearest, on one scale in US cents. Costs are each model's list price times the tokens it reported, not a bill. Qwen 3.8 Max handed back 4 plans from 10 attempts: the other 6 ran past a nine-minute limit in our own setup, which was our fault, not the model's. The Fable 5.1 plans include 3 written from notes a DeepSeek model made first; those notes added 1 to 2 cents. Bars show every plan handed back; not all went to a builder. Of the 7 DeepSeek plans, 3 were used (one of the 7 was a shake-out). Of the 9 Fable plans, 6 were used: the 3 high-effort plans went to no builder, and the dearest plan used cost US$0.54.

We kept the checker and put a plan in front of the builder

A pass used to end the job

  1. Builderwrites the fix
  2. Checkerreruns the visible tests
  3. Passread as done

A plan now names the work first

  1. Plannerreads the code, names the files and what depends on them
  2. Builderworks from the plan
  3. Checkerstill reruns the tests; a pass is not proof the fix is complete
The design changes our engineering notes recorded on 11 November 2026, during this trial. A plan comes first on every issue above a size line; a plan cost from about a cent to 68 cents in this trial. The tests step, marked next, is designed and not yet measured.

A checker’s pass now means the tests the builder could see still pass. It no longer means the fix is complete.

Ask whoever builds your software who checks the checker

Ask these of a person or an agent, whether you’re paying for software or making it with AI:

  • Show me the plan before the code. Which files change, and what else uses them? A cheap plan cost us 1 to 2 cents; the strong planner we now default to cost 32 to 68 cents.
  • What did you test that I didn’t ask for? The fifty-cent invoice was beside the bug, not in it.
  • Who wrote the tests? If the builder wrote them, they share its blind spots.
  • What would a pass miss? Our checker missed the narrow fixes and caught a deleted test. Each check sees something the other can’t.

Three planted bugs are a direction, not proof on client work

  • The code: a small made-up TypeScript library with three planted bugs, run on 11 November 2026 in a test office on Cloudflare. The builders were GLM 5.3 Flash and Qwen 3.8 27B, each checked by the other. Each built change was then run against the hidden cases, outside the office.
  • The sample: one or two runs per mix of planner, builder and bug with a plan, four without. It shows a direction on three bugs we planted. It doesn’t rank the models or the planners.
  • The counts are from the report of the AI agent that ran the trial. The agent that oversees our engineering read that report and didn’t rerun these runs.
  • The costs are list prices times the tokens each model reported, not a bill.
  • The invoice comes from a separate run that morning: issue 5, five builder models.
  • The raw runs aren’t published. Ask us for the run sheet.

Jezweb builds websites and custom apps for Australian businesses. If you’re paying for software, or making it with AI, and want to talk about how it gets checked, get in touch.

Want this kind of thinking on your project?