Two clean answers, one of them counterintuitive. GLM 5.2 can't run on your Mac — not close. And a bunch of 7Bs collaborating is a known trap — but there's a version of the idea that's genuinely good, and your gate is the secret weapon that makes it cheap.
GLM 5.2 is a 743B-param sparse MoE (256 experts, DeepSeek-V3.2-style attention). The 4-bit MLX build is 418 GB on disk and is described as "tight on 512 GB" RAM. Even the most compressed mixed 3.6-bit variant needs ~360 GB. It is a frontier-scale model — runnable on a 512 GB Mac Studio Ultra or a cluster, not a 24.5 GB box. Off the table locally.
"Near-Opus, but local" and "fits in 24.5 GB" are, in mid-2026, mutually exclusive. The models that rival frontier APIs are all 400B–1T params — they need 256–512 GB. On your hardware, local LLMs are a different weight class: very good for bounded, verifiable work, not a drop-in Opus replacement. That's exactly why the offload tool's wedge is privacy + the easy/medium slice of work, not "replace Claude on hard reasoning." The router should embrace that, not fight it.
The current sweet spot for 24 GB hasn't moved tiers, but the generation has — your ladder is on Qwen 2.5; the 3.6 line is out and meaningfully better at code.
| Tier | Current (2.5 gen) | 2026 upgrade (3.6 gen) | Fit @ 24.5 GB | Note |
|---|---|---|---|---|
| FAST | Qwen2.5-Coder-7B | Qwen3-7B | ✓ ~4 GB | HumanEval 76 — best under 8B |
| MID | Qwen2.5-Coder-14B | Qwen3.6-27B dense Q4 | △ tight | SWE-bench 77.2% — strong |
| QUALITY | Qwen3-Coder-30B-A3B | Qwen3.6-35B-A3B MoE 4-bit | ✓ ~17–18 GB | 30B knowledge, 3B speed |
Not urgent, but if we're touching routing anyway, pulling the rungs up a generation is low-risk and free quality. Worth a quick eval pass before committing.
This is the juicy one, and the literature has a real plot twist. Here's the honest arc:
Together's MoA showed multiple open models, layered (proposers → aggregator, each layer reading the previous layer's answers), scoring 65.1% on AlpacaEval 2.0 vs GPT-4o's 57.5%. Collaboration genuinely lifts quality — models write better answers when they can see peers' attempts.
The catch: MoA's proposers were 110B-class models. The result is "big diverse models collaborate," not "small models collaborate."
A follow-up asked: is mixing different models actually the point? They found MoA is highly sensitive to proposer quality — there's a quality–diversity trade-off, and chasing diversity drags in weak models that drop the result. Their Self-MoA (sample the single best model many times and aggregate) beat mixed-MoA by 6.6% on AlpacaEval and +3.8% across MMLU/CRUX/MATH.
Three diverse 7Bs voting tends to do worse than sampling your one best model three times. Diversity only pays when every voice is already strong. Your strongest fit-able voice is the 30B/35B — not a committee of 7Bs.
Best-of-N sampling: generate N candidates, a verifier picks the best. The research uses learned verifiers — but you already have a perfect one for code: the gate. py_compile, a test command, a JSON-schema check — an objective, free oracle. So the pragmatic "ensemble" is: sample N candidates, run each through the gate, return the first that passes. No aggregator LLM, no voting heuristics — the gate is the selector, and it's already built.
Short answer: not iterate — it's the wrong shape — but yes, as a sibling tool.
| Pattern | Control flow | Tool |
|---|---|---|
iterate (today) | Sequential cascade: try → gate → retry/climb → escalate. One line of attempts. | exists |
| Best-of-N "council" | Fan out N candidates → gate-filter → return a winner (or escalate if none pass). | new sibling |
The warm pool from v2 makes the substrate cheap: three 7Bs co-reside in ~12 GB, so a council holds its members hot with no per-call load. But the v2 caveat bites harder here — one GPU serializes the N inferences (and MLX's concurrent-model threading still crashes), so a council is a latency-for-quality trade, queued, not parallel. It's coordinatable; it just isn't free, and it isn't fast.
A council of 7Bs sounds cool and democratic. Build it.
Self-MoA says it's the wrong cool. Mixing weak models underperforms sampling one strong model. If we want N opinions, sample the 30B N times, not three 7Bs once. "Bunch of 7Bs" loses to "one good model, a few times."
Then sample the 30B N times. Best-of-N, gate picks the winner. Ship it as the headline feature.
Careful — that's N× latency on one GPU, and iterate already gets most of the benefit by feeding gate failures back in for a guided retry. Best-of-N is blind resampling; iterate's retry is informed. For code with a compile/test gate, informed retry usually beats blind N-sampling per unit of latency. Best-of-N wins when failures are about unlucky sampling, not misunderstanding — and we don't yet know which we have.
So all this research and the answer is "don't build the councils"? Anticlimactic.
The answer is "we now know what not to build, and why" — that's worth a lot. GLM-class local is a hardware fantasy on 24 GB; weak-model councils are a Self-MoA trap; blind best-of-N is a latency tax that informed retry often beats. What survives is sharp: refresh the ladder a generation, keep the warm-pool eager router, and hold best-of-N as a measured experiment gated on real single-shot failure rates. Negative results that stop us wasting weeks are the most valuable output of a spar.
One genuinely new keeper, though: the gate-as-verifier framing. We built the gate for self-correction; it doubles as a free best-of-N selector the day we want one. That's latent capability worth writing down.
Unchanged core: warm-pool eager routing (v2). New from v3:
council tool, gate-as-verifier, justified only if logs show high single-shot failure rates that informed retry isn't already fixing.council tool (my lean), or do you want it prototyped now alongside the router so you can feel the latency/quality trade yourself?7B+14B vs lazy/TTL), trust-oMLX-eviction vs our own, and what drives "hard → go big" (gate type vs category vs both).