You asked: could we keep several small models warm at once, and only evict to load the big one? And could that spawn parallel local agents? Short version: the warm pool is right and changes the design; the parallel-agents part is half-right and I need to set expectations.
oMLX keeps multiple models resident and switches between them on demand; the only ceiling is total unified memory (~24.5 GB on your box). That means switching among the small/mid models is effectively free — they're already in memory. The cold-load cost only appears when we need the 30B, because it's the one model big enough to require kicking something out.
| Combination | Footprint | Fits under ~24.5 GB? | Leaves for context/KV |
|---|---|---|---|
7B + 14B | ~12 GB | ✓ comfortably | ~12 GB |
7B + 14B + DeepSeek-Lite | ~20 GB | ✓ tight | ~4 GB |
7B + 30B | ~21 GB | ✓ (fast + quality both warm!) | ~3 GB |
14B + 30B | ~25 GB | ✗ over ceiling | — |
30B alone | ~17 GB | ✓ | ~7 GB |
The 30B is the only model that forces an eviction decision, and it can still co-reside with the 7B. So a workable steady state is: keep 7B + 14B warm for everyday work (free routing between them); when a task is judged hard, evict the 14B to bring the 30B in alongside the 7B. The eager-swap you picked now costs ~nothing for the easy/medium tiers and only pays a load exactly when it's worth it — going to QUALITY.
Holding models warm ≠ running them in parallel. Two hard facts:
Verdict: the parallel-agents idea is a great programming model (fan out subtasks, queue them, overlap I/O) but not a speedup, and the threading isn't stable yet. Treat warm-pool as a latency win (no cold loads), not a throughput win — and serialize GPU access behind a queue rather than firing concurrent inferences.
If small models are free to switch, why route at all? Just always start on the 7B, let the gate climb to 14B (free), then 30B (one load). The cascade already does this and never guesses wrong.
Closer than before — with a warm pool the cascade is cheap. But two leaks remain. (1) The 30B load is still seconds; for a task we know is hard, eating two failed small rounds first is pure latency waste — eager routing skips straight to it. (2) The 30B at ~51 tok/s isn't even the slowest; the 14B is ~28 tok/s. Starting a trivial stub on the 14B is slower than the 7B at ~80. Routing trims both ends; the warm pool just makes the trimming nearly free.
You want to pre-warm 7B+14B. That's ~12 GB sitting resident, eating context headroom, even for a user who only ever does easy tasks. Wasteful.
It's a TTL/keep-alive policy, not a permanent pin. Warm the pool lazily — load a tier the first time it's routed to, keep it warm with an idle TTL, evict LRU when memory pressure demands (i.e., when the 30B needs room). That's exactly how Ollama's keep-alive works. Steady state adapts to the actual workload instead of pinning a guess.
Now the router has to track what's resident, memory budgets, eviction order, TTLs. That's a little memory manager. You've turned a routing tweak into a subsystem. YAGNI?
Legit risk. So lean on oMLX — it already manages the engine pool and eviction. We don't rebuild that. Our job is only: (a) pick the entry model from the task signal, (b) ask oMLX to ensure it's loaded, (c) let the existing cascade climb. The "memory manager" is one health-pool read, not a new component. If oMLX's own eviction is good enough, we add nothing; if it thrashes, then we add a hint. Measure first.
And the parallel angle isn't wasted — it just changes shape. Instead of "run 3 models at once," it's "a warm pool lets a future multi-agent workflow hand different subtasks to different already-loaded models without paying a load each time, served through one GPU queue." That's a real capability for later — just don't sell it as concurrency speedup.
A trained classifier, a custom memory manager, and true parallel multi-model inference. All three are solving problems we either don't have yet or can't safely build today. Revisit when data or a fixed MLX says otherwise.
7B + 14B (everyday, ~12 GB), or 7B + 30B (fast + quality always hot, skip 14B, ~21 GB but thin on context)? Or purely lazy — warm nothing, load each tier on first route and let TTL decide?