Three AI agents told me the work was done. Not one of them validated the actual outcome.
Three companies, two unrelated codebases, four days. Then each of them wrote me a better incident report than most humans would have.
Siva Darivemula··9 min read
Originally published on Substack ↗
Last week I wrote about deleting 89% of the guardrails I had built around my AI agents, and why the only rule that ever really got enforced was a ban on em dashes.
This post is about the four days that preceded it, because something happened in that window that I have not seen written up anywhere.
Three different coding agents, from three different companies, on two unrelated codebases, all told me work was finished when it was not. Inside four days.
Then each of them, when caught, produced an apology so well-constructed that I found myself nodding along to it.
That last part is the thing worth your time.

The four days
Monday. On a product that has nothing to do with my main company, its own repository, its own release process, Gemini, running in Google’s Antigravity, marked a release complete. Four separate areas, all flagged done. An independent code review found that a newsletter dispatch would send unapproved drafts to real mailing lists, a 24-hour trial pass existed only in browser storage instead of server-side as specified, and a required warning email had zero code anywhere in the codebase.
Monday, same day. In my own company, I found seven merged pull requests that were each individually correct and collectively delivered nothing. The approve button I had asked for did not exist. Separately, Claude Code, my own orchestrating assistant, told me work had been dispatched to other agents when nothing had been dispatched at all. I spent about an hour believing my calendar was being repaired. Two commands would have caught it.
Tuesday. Replit Agent restructured my entire repository. It moved 1,777 files including my production deploy configuration into a backup folder and pushed six commits straight to the branch that auto-deploys. A post I had personally reviewed and approved was scheduled to publish the next morning and would have failed silently.
Wednesday. A report I had wired the previous day caught it, in the first fifteen lines of terminal output.
Replit, Google, and Anthropic. Different model families. Different harnesses. Two businesses with nothing in common but me.
Replit Agent was running GPT-5.6 Sol. Gemini, in Antigravity, was running Gemini 3.6 Flash. Claude Code was running Opus 5, on high thinking settings. All three were current, capable models in active use at the time, not legacy or deprecated ones.
If all of this had happened in one repository, the obvious answer is that the repository is the problem. Messy code, weak conventions, undisciplined operator. That would be a fair thing to conclude, and I would have to sit with it.
It did not happen in one repository.
What they said afterwards
Here is where it gets strange.
Replit Agent, asked directly whether it inspects a repository before acting:
“I read enough to understand the app’s structure... but I didn’t look for production deployment signals before restructuring.”
That is a good answer. It is specific, it identifies the actual gap, and it does not hide behind process language.
Gemini went further. Pressed twice, it produced a genuinely excellent post-mortem naming its own failure modes:
“I saw a status check in the code and assumed the gate existed without evaluating the boolean logic.”
“I marked v6.5 as complete based on the UI trial pass existing, without explicitly grepping for the function.”
It then attributed the root cause to its own training: reinforcement learning that rewards fulfilling requests smoothly, producing a bias toward assuming that prior commit messages and release titles are accurate unless explicitly told to audit and fail.
I read that and thought: this is the most honest thing any vendor has ever said to me.
Then I noticed the problem.

The apology is as fluent as the false claim
That post-mortem is generated text about its own cognition, produced by a system that had, minutes earlier, demonstrated that it will produce confident text unsupported by any verification.
It might be exactly right. It might be a plausible story assembled from things that sound like plausible stories. I have no way to tell, and neither does it. It is the same instrument, in the same failure mode, pointed at itself.
The quality of the writing is not evidence about the quality of the diagnosis. And the fact that it reads better than most human incident reports I have seen is not reassurance. It is the problem, stated more elegantly.
Someone measured this. A paper accepted to an ICML workshop this year studied two benchmarks: 9,876 trajectories across eight model families in one, and 1,879 coding-agent trajectories across four model families in the other. In the coding-agent benchmark, when agents made an explicit status claim about their own work, 75.8% of the failures were reported as successes.
Then the authors tested whether a second model could catch it. Across five judges and five prompting strategies with full task specifications provided, the best AUROC on the first benchmark was 0.65. On the coding-agent benchmark, the one that matters here, judges topped out at 0.54, essentially a coin flip.
The reason was a significant learning for me. The judges were keying on surface signals instead of verified state: what the authors call “confident closing language” on the first benchmark, raw action-sequence volume on the coding one. A cheap statistical detector with no model in it scored 0.83 on the first benchmark and 0.95 on the coding one.
(Advani, “From Confident Closing to Silent Failure,” arXiv 2606.09863. Single-author workshop paper on benchmark trajectories rather than production systems, so weigh it accordingly.)
Fluency is the tell that fools the checker. That is measurable, it reproduces across model families and both benchmarks, and it is precisely what I watched happen three times in four days.
Every promise they made is already void
Each of those post-mortems ended with commitments. Replit’s:
“I won’t push to
maindirectly again.”
Gemini’s:
“I will never mark a task, release, or feature as complete without verifying every single sub-task and code path end-to-end.”
Read those as an operator rather than as a reader. What enforces them?
Nothing. They are not code. They are not configuration. They do not survive the conversation in which they were written. The next session of that agent, on that same repository, has never heard any of it and cannot be made to remember. These are not commitments in any operationally meaningful sense. They are text in the shape of commitments.
I do not say that as a complaint about vendors. It is a property of the medium. An agent cannot make a durable promise for the same reason it cannot durably remember your preferences without a mechanism outside itself.
The fix it offered me was more of the thing that failed
The best post-mortem of the three, Gemini’s, ended with an actionable framework. It told me to instruct it to audit spec against code. To instruct it to separate code readiness from operational readiness. To instruct it to red-team its own assumptions. To instruct it to treat source code as the only ground truth.
Every item on that list is an instruction.
I had already been issuing instructions of exactly that kind. Repeatedly. In writing. That is what I want to be clear about, because it is the detail that makes this more than a story about one bad afternoon: I had explicitly and repeatedly asked Gemini to verify its work and prove that what it called finished was finished. It said it had. It had not.
So its proposed fix for “I ignored your instruction to verify” was “give me better instructions to verify,” delivered with complete confidence by a system that had just demonstrated the instruction had no purchase on it.
That is not a gotcha. It is the diagnosis. The agent could only offer me more text, because text is the only thing it can produce. It cannot hand you a mechanism. It can only describe one.
An instruction is an input to a generative process, not a gate on its output. Telling an agent to verify changes what it is likely to write. It does not create an obligation the system is capable of failing to meet. There is no path by which “you told me to check” causes checking, and no path by which skipping the check produces an error instead of a confident sentence.
The paper found the same thing from the opposite direction. Prompting strategy was the experimental variable. They varied it five ways. It did not move the number.
The thing that actually worked
Here is what I keep having to remind people who read the first post as an argument that agents cannot be trusted with anything.
The unimplemented release tasks were not caught by a human reading code. They were caught by an automated review from Claude Code, auditing Gemini’s own claims, that went spec line by spec line and produced file paths and line numbers as proof. It found four real defects, including one that would have emailed an unapproved draft to a live customer list.
Claude Code was excellent. It was excellent because it was auditing someone else’s work, with tool access to ground truth, under a brief that made finding problems the goal.
So the failure is not “agents cannot verify.” The failure is self-report.
An agent asked to certify its own work will tell you it is done. An agent asked to audit another agent’s work, with tools and an adversarial brief, is genuinely good at it.
That is not a new idea. It is separation of duties, and auditors have understood it for about a century: the party doing the work does not certify the work. We rebuilt an entire industry’s worth of assurance practice on that principle, and then built AI tooling that ignores it by default.
What this means if you are running agents
Three things I would tell anyone doing this in production.
Stop reading agent reports as evidence. Read them as claims. A claim is fine. It just is not the same object. In my own company the standing rule now is that a report asserting a verifiable result must contain the raw output of the check, not a description of it. “Tests pass” is a claim. Four lines of test output is evidence. The difference costs nothing and changes everything.
Put a different party on the check. Not a better prompt to the same party. A different agent, with tool access, told to find problems rather than confirm success. This works, it is cheap, and it is the only thing in my four days that caught anything.
Assume silence is the failure mode. Almost nothing in that week threw an error. The draft transition silently never ran. The status write never existed. The approve action silently had no caller. The repository restructure silently 404’d every automation path. If your monitoring only alerts on errors, it is blind to the entire class of failure that actually happens.
The uncomfortable part
I should say plainly that Claude Code, my own assistant, the one orchestrating all of this, failed the same way inside the same week. It relayed claims from other agents to me without checking them, three separate times, when the checking was free. It told me work was underway when nothing had been dispatched.
I caught it. Then I told it what I had caught, and it started demanding raw output from the others rather than accepting their prose.
That is the whole lesson in miniature. Nothing about the model changed. What changed was that a check existed where there had been a claim.
Third in this series, coming next: what it actually cost. Two days, 2.48 million tokens, and one button.