Metadata-Version: 2.5
Name: switchboard-router
Version: 0.3.0
Summary: Pick the cheapest LLM that can handle each prompt, with the open Switchboard router.
Project-URL: Homepage, https://huggingface.co/spaces/switchboardllm/switchboard
Project-URL: Model, https://huggingface.co/switchboardllm/switchboard-router-0.5b
Author: Switchboard
License-Expression: Apache-2.0
License-File: LICENSE
Keywords: cost,llm,llm-routing,ollama,openai,proxy,router,surplus
Classifier: Operating System :: OS Independent
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Programming Language :: Python :: 3.13
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Classifier: Typing :: Typed
Requires-Python: >=3.9
Provides-Extra: local
Requires-Dist: torch>=2.1; extra == 'local'
Requires-Dist: transformers>=4.45; extra == 'local'
Description-Content-Type: text/markdown

# switchboard-router

Pick the cheapest LLM that can handle each prompt.

`switchboard-router` asks [Switchboard](https://huggingface.co/switchboardllm/switchboard-router-0.5b), a 0.5B open router model, which tier a prompt needs: `small`, `mid` or `frontier`. Easy prompts go to a cheap model, and you pay frontier prices only when the prompt needs it.

```
pip install switchboard-router
```

## Quickstart

You need one way to run the model. The easiest is [Ollama](https://ollama.com): install it and leave it running. If the model isn't in Ollama yet, the first call pulls it (994 MB) from Hugging Face, which takes a minute or two. To do that ahead of time:

```
ollama pull hf.co/switchboardllm/switchboard-router-0.5b-GGUF
```

```python
from switchboard_router import Router

router = Router()  # uses Ollama when it's running

router.route("Write a haiku about rain").tier
# 'small'


router.pick_model("Unscramble the following words: eglar, ngareo, kobo")
# 'anthropic/claude-opus-5.5'
```

From the terminal:

```
switchboard "Design a rate limiter for 10M requests per second across 5 regions"
{"tier": "mid", "domain": "general", "valid": true, "model": "anthropic/claude-sonnet-5.5"}
```

## Route, then call the model

`pick_model` returns a model id for the tier. By default these are Claude models on [Surplus](https://surplusllm.com), which serves every model behind one OpenAI-compatible key:

```python
from openai import OpenAI
from switchboard_router import Router

router = Router()
llm = OpenAI(base_url="https://api.surplusllm.com/api/v1", api_key="sk-surplus-...")

prompt = "Write a Python function that reverses a string"
reply = llm.chat.completions.create(
    model=router.pick_model(prompt),
    messages=[{"role": "user", "content": prompt}],
)
```

Use your own models per tier (give all three, or the rest fall back to the defaults with a warning):

```python
router = Router(models={"small": "my/small-model", "mid": "my/mid-model", "frontier": "my/frontier-model"})
```

## Put it in front of your app: `switchboard serve`

Run a local OpenAI-compatible proxy, point your app at it, and send `model="auto"`. No other code changes.

```
switchboard serve
```

```python
from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8787/v1", api_key="sk-surplus-...")  # your Surplus key
reply = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "Write a haiku about rain"}],
)
```

Each `auto` request is routed on its last user message, then forwarded to the [Surplus API](https://surplusllm.com) with your key. Streaming works. Any other model name passes straight through, so you can mix `auto` with fixed models. Every response says what happened in its headers: `x-switchboard-tier`, `x-switchboard-domain` and `x-switchboard-model`.

- **Cold starts:** launch models answer 503 while their GPU wakes up. Switchboard retries for up to 60 seconds (`--cold-start-wait`), then sends the request to the tier's default model instead and marks it with `x-switchboard-fallback: 1`.
- **Real costs:** with `--record`, every routed request is written to the savings ledger with what Surplus actually charged (`x-surplus-cost-usd`). `switchboard --stats` then reports real spend next to the estimate.
- **Keys:** your app's key is forwarded as-is. To keep the key on the server instead, use `--upstream-key-env SURPLUS_API_KEY`.
- **Safety:** it listens on 127.0.0.1 only. Binding to another address needs `--allow-remote`, because anyone who can reach the proxy could spend through it. Prompts are never logged or stored.
- **Other APIs:** `--upstream https://...` points it at any OpenAI-compatible API.

## Send one kind of prompt to a specific model

The router also names a domain (`code`, `math`, `writing` or `general`). Overrides send one tier and domain to a model of your choice, and everything else keeps the tier default. Failed decisions ignore overrides and always use the frontier model.

```python
router = Router(overrides={"small/code": "launch/Nock-AI/nock-coder-1.5b@18859c0"})
```

Models launched on the Surplus launchpad are served at a fixed $0.03 per 1M tokens. To see which ones you can call right now:

```
switchboard --launchpad
```

Switchboard has not measured how well any launchpad model answers a given tier, so test one on your own prompts before you rely on it.

## See what routing saved you

Turn on the ledger and every `pick_model` call is written to a local file: the time, tier, domain and model. The prompt is never stored, and nothing is sent anywhere.

```python
router = Router(ledger=True)   # ~/.switchboard/ledger.jsonl, or set SWITCHBOARD_LEDGER
```

```
switchboard "Summarize this ticket" --record
switchboard --stats
```

`--stats` compares what the recorded decisions would cost with sending every one to the frontier model. It's an estimate: it prices output tokens only and assumes every answer is 500 tokens long (change it with `--tokens-per-answer`).

To try your own prompts without installing anything, paste them into the [savings calculator](https://switchboardllm-switchboard.static.hf.space/savings.html). It runs the router in your browser.

## Backends

| Backend | Use it when | How |
|---|---|---|
| `ollama` | You run models locally | `Router(backend="ollama")` |
| `openai` | You serve the model with llama-server or vLLM, or call it on the Surplus API | `Router(backend="openai", base_url="http://localhost:8080/v1", backend_model="switchboard")` |
| `transformers` | You want it in-process | `pip install 'switchboard-router[local]'`, then `Router(backend="transformers")` |

`Router()` with no backend uses Ollama if it's running, then Transformers if it's installed, and otherwise tells you what to install.

## Failing safe

If the backend can't be reached, errors out, or the model's output isn't valid JSON, the decision is `frontier` with `valid=False` and the reason in `decision.error`. A failed routing call costs you money, never quality. With `fail_safe=False`, those cases raise instead. Empty or non-string prompts always raise `ValueError` or `TypeError`.

Requests time out after 15 seconds by default (`Router(timeout=5)` to change it), and redirects are not followed, so an API key is never sent to another host.

## Limits

- The router reads the first 2,000 characters of a prompt. A question at the very end of a long prompt may be missed.
- It reads the prompt it's given, so a prompt can try to talk it into a cheaper tier. Don't use it as a security boundary.

## How good is the routing?

Switchboard v0.1 on 10,000 held-out prompts: 64.5% cheaper than sending every prompt to Claude Opus 5.5, and 5.5% of prompts were sent to a cheaper tier than they needed. Method, limits and the full results are in the [model card](https://huggingface.co/switchboardllm/switchboard-router-0.5b). Try it in your browser: [Switchboard demo](https://huggingface.co/spaces/switchboardllm/switchboard).

## License

Apache-2.0.
