Eval Results5 models × 24 gated cases2026-06-17

Local Model Eval — Results

Which installed model wins per task, measured on your machine. Quality = gate pass-rate · Speed = median latency · Cost = mean output tokens.

Headline: on bounded easy/medium work, every coding model passes nearly everything — so the differentiator is speed, not quality. And the surprise: the 30B MoE is as fast as the tiny model while having the most headroom. The reasoning model and the 14B are the losers.

The ladder I'd pick

recommended Two rungs cover the offload workload:

Default / quality rung → Qwen3-Coder-30B-A3B. 24/24 and ~650ms median — it's a Mixture-of-Experts (only 3B active per token), so it runs at small-model speed while holding 30B-class knowledge. Best of both; make it the everyday model.
Turbo floor → DeepSeek-Coder-V2-Lite. 24/24 at ~540ms, the fastest. Ideal for trivial boilerplate/extract bursts where you want instant.
Drop the 14B from the offload path. 24/24 but the slowest of the capable group (~1.7s) with no quality edge on this suite. It earns its place only if harder cases later separate it from the 7B.
Reserve the 27B Opus-Distilled for genuine reasoning, not offload. ~30s median is 45× the 30B — unusable for quick tasks.

Per-model summary

Model	Pass	Median latency	Mean tokens
DeepSeek-Coder-V2-Lite (turbo)	24/24	544ms	37
Qwen2.5-Coder-7B (fast)	24/24	922ms	27
Qwen2.5-Coder-14B (everyday)	24/24	1.7s	28
Qwen3-Coder-30B-A3B (quality, MoE)	24/24	649ms	28
Qwen3.5-27B Opus-Distilled (reasoning)	22/24	29.5s	264

Pass = gate pass-rate across all 24 cases. Latency = median wall-clock per case. Tokens = mean completion tokens (cost/verbosity proxy).

Quality heatmap — gate pass-rate by category

Model	boilerplate	summarize	extract	review	explain	codegen
DeepSeek-Coder-V2-Lite (turbo)	4/4	4/4	4/4	4/4	4/4	4/4
Qwen2.5-Coder-7B (fast)	4/4	4/4	4/4	4/4	4/4	4/4
Qwen2.5-Coder-14B (everyday)	4/4	4/4	4/4	4/4	4/4	4/4
Qwen3-Coder-30B-A3B (quality, MoE)	4/4	4/4	4/4	4/4	4/4	4/4
Qwen3.5-27B Opus-Distilled (reasoning)	4/4	4/4	4/4	4/4	4/4	2/4

Hover a cell for its latency + tokens. Only the reasoning model dropped cases — and those were gate failures: it wraps code in chain-of-thought prose, so the whole-output compile check fails.

Speed heatmap — median latency by category

Model	boilerplate	summarize	extract	review	explain	codegen
DeepSeek-Coder-V2-Lite (turbo)	1107.5ms	496.0ms	432.0ms	327.0ms	583.0ms	950.0ms
Qwen2.5-Coder-7B (fast)	1.9s	831.5ms	684.0ms	315.0ms	1273.5ms	1242.0ms
Qwen2.5-Coder-14B (everyday)	4.2s	1.7s	1121.0ms	482.5ms	2.4s	2.2s
Qwen3-Coder-30B-A3B (quality, MoE)	917.5ms	647.0ms	516.5ms	372.0ms	677.5ms	1155.5ms
Qwen3.5-27B Opus-Distilled (reasoning)	20.3s	26.7s	26.5s	41.1s	33.4s	35.8s

Green <1.5s · amber <6s · red ≥6s. The 27B reasoning model is off-the-charts slow (20–41s per case).

The operational finding (matters for the product)

You cannot sweep all 5 models in one pool — the warm pool OOMs.

The first full sweep loaded the small models first; by the time the big ones were requested, the engine pool was full and they returned 507 Insufficient Storage (and the 14B got knocked over with 400/connection errors under the pressure). The harness's resilience held — it recorded failures instead of crashing — and clean numbers came from running each big model in isolation (one big model at a time fits; two ~17GB models do not, under the ~24.5GB ceiling).

This is the exact warm-pool/eviction problem the routing design flagged — now proven empirically on your hardware. It directly shapes the deferred auto-ladder selector: it must evict before loading a big model, and never try to co-resident two big ones. A practical sweep harness needs a "big models run solo" mode.

Caveats — read before trusting the ladder

This suite is easy/medium. Every model passing ~everything means the suite doesn't yet separate them on quality — only speed. Harder cases (multi-step refactors, tricky debugging) would likely spread the field and justify the 30B/14B over the 7B/turbo. That's what Phase 2 capture + weekly review is for: grow the suite with real, harder prompts.
The 27B's 2 misses are gate artifacts, not wrongness. Its answers were likely correct but wrapped in reasoning prose; a code-extraction step (not just fence-stripping) would recover them. Don't read 22/24 as "lower quality."
Latency is single-run median, warm model. Cold-load (first call after a swap) adds seconds and isn't in these numbers.

Next steps

Set Qwen3-Coder-30B-A3B as the default offload model and DeepSeek-Lite as the turbo floor; retire the 14B from the offload path.
Grafana now shows this consolidated run in the "Model Eval" row (5 models, latest run).
Phase 2: capture real prompts → weekly review → grow the suite with harder cases so quality (not just speed) gets measured.
Auto-ladder selector must encode the one-big-model-at-a-time rule this eval just proved.

Generated from ~/.omlx/eval-results.jsonl · 5 models × 24 gated cases across 6 categories · big models evaluated in isolation due to the memory ceiling. Counts/labels only — no prompt or output text stored.