iterate ladder by task — without paying a model-swap tax. · 2026-06-17Right now iterate always starts on whatever model is loaded, burns up to 3 rounds, then climbs. A one-line docstring and a gnarly refactor get the same opening move. This is about choosing the starting rung from the task — and the wrinkle that makes our case different from every routing paper out there.
I read the current routing work — RouteLLM, the cascade surveys, the cost/quality routers. The dominant pattern is a cascade: start on the cheapest model, check quality, escalate only on failure. Teams report 30–85% savings doing exactly this. Our iterate ladder already is a cascade. Good — we're on the right track.
In the cloud, GPT-4 and Mixtral are both resident on someone else's hardware. Routing is pure prediction — picking which always-warm endpoint to hit costs nothing. On your Mac, only one big model fits under ~24.5 GB at a time. Switching from the 14B to the 30B means unload + cold-load 17 GB — seconds of wall-clock, every time. So our routing problem isn't "which model is best." It's "which model is best net of the swap tax." That reframes everything below.
| Cloud router (the papers) | Our local box | |
|---|---|---|
| Cost unit | $ per token | Wall-clock seconds (tokens are free) |
| Switching models | ≈ 0 (all resident) | Seconds of cold load (one big model at a time) |
| Main waste to kill | Frontier $ spent on easy queries | Doomed rounds + unnecessary swaps |
| Best move | Predict hard→big up front | Predict and minimize thrash; prefer the resident model when it's good enough |
So a naive "task-aware" router copied from a blog post could make us slower: classify → decide "hard" → swap to 30B for a task the resident 14B would've nailed in one round. The design has to treat a swap as a first-class cost, not a free lunch.
Why build routing at all? The cascade self-corrects. If the small model fails the gate, we climb. The worst case is a few wasted rounds.
Because "a few wasted rounds" is the whole latency budget. A hard task on the 14B fails the gate 3× (slow tokens, no chance), then we swap and run the 30B. We paid for 3 doomed rounds before doing the thing that was always going to work. And the other tail: a trivial boilerplate stub starts on the 14B default when the 135-tok/s tiny model would've one-shot it. Task-awareness trims both tails.
Fine, but a pre-classifier that's wrong is worse than the cascade. Misjudge "hard," swap to the 30B, and you've eaten a 17 GB cold load for a task the 7B could do. At least the cascade only climbs when it has proof (a real gate failure).
That's the real risk, and it's the argument against an aggressive up-front classifier. The cascade's strength is that it escalates on evidence, not a guess. So the design shouldn't replace the cascade — it should pick a smarter entry point and keep the cascade as the correctness net underneath. Get the floor wrong, the gate still catches it.
Then where does the routing signal even come from? You're not training a RouteLLM classifier on a single-user box with no labeled data.
Right — that's overkill and we have no training set. But we already pass a category on every call (boilerplate / summarize / extract / review / explain / other) and we know the gate type. Boilerplate with a min_len gate is easy; "other" with an executable check_command that must compile is hard. That's a free, structured signal we're currently throwing away. Start there.
Category is agent-supplied. I'll be lazy and tag everything other. Your table collapses to "always start mid."
Fair hit. Two answers. One: other should map to the current behavior (start on resident), so laziness degrades to today, never worse. Two — and this is the honest move — we already log winning_rung per category in the call log. Before we hard-code a routing table, we measure: for each category, what rung actually won? If boilerplate already passes on rung 1 every time, routing it down is safe and proven. If other is noise, we learn the signal is weak and don't over-invest. Instrument first, route second.
And the swap tax gives us a cheap dominant heuristic regardless of category: if the routed tier equals the resident model, never swap. Stay sticky. A run of easy tasks keeps the small model warm; we only pay a load when we genuinely cross a tier boundary. That alone removes most thrash.
Rules over free features: prompt length, code-fence presence, keywords (refactor/boilerplate/summarize → easy; architect/debug/why → hard), gate type, category.
Ask the always-resident tiny model for a difficulty label before picking the worker model.
Map the category we already collect to a starting tier + round budget. No new machinery.
C as the base, A as a one-rule nudge, cascade as the net, sticky-resident to kill thrash. Category picks a tier + round budget; one structural nudge (executable gate ⇒ bump a tier, since code that must compile is hard for tiny models); never swap if routed tier == resident; the gate-driven cascade still runs underneath as the guarantee.
It only earns its 200–500 ms if the cheap heuristic is wrong often. We don't know that yet. So ship D, keep logging winning_rung by category, and revisit B only if the data shows the floor is mis-picked enough to hurt. Don't pay for a classifier to solve a problem we haven't measured.
Routing tiers map to models known to fit, one big one at a time. Tiers are an ordering, not simultaneous residents:
1Measure, don't guess. Add a tiny call-log analysis: for each category, what winning_rung distribution do we actually see? This tells us which categories are safe to route down and whether the signal is real. (We have the data today.)
2Routing table from evidence. Encode category → entry tier + max rounds, justified by step 1, with other ⇒ today's behavior so the change is strictly non-regressive.
3Swap-aware execution. Sticky-resident check; only swap on a genuine tier crossing; the existing one-rung climb design respects the ceiling.
4Log the routing decision. Add entry_tier + swapped to the call log so the dashboard can show whether routing is paying off (and feed a future B decision).
category (agent-supplied, free) or with the gate type (objective, but only present when you pass a gate)? Or weight both?