Metadata-Version: 2.4
Name: gpuroutertest
Version: 0.4.1
Summary: Launch, load-test, and tear down a GPU-backed HTTP API on your own AWS account with one command.
Author: Susmit Kulkarni
License: MIT
Project-URL: Homepage, https://github.com/HolboxAI/gpuroutertest
Project-URL: Issues, https://github.com/HolboxAI/gpuroutertest/issues
Keywords: aws,ec2,gpu,api,deploy,cli,benchmark,load-testing
Classifier: Development Status :: 4 - Beta
Classifier: Environment :: Console
Classifier: Intended Audience :: Developers
Classifier: License :: OSI Approved :: MIT License
Classifier: Programming Language :: Python :: 3
Classifier: Programming Language :: Python :: 3.9
Classifier: Programming Language :: Python :: 3.10
Classifier: Programming Language :: Python :: 3.11
Classifier: Programming Language :: Python :: 3.12
Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
Requires-Python: >=3.9
Description-Content-Type: text/markdown
License-File: LICENSE
Requires-Dist: boto3>=1.26
Requires-Dist: typer>=0.9
Provides-Extra: benchmark
Requires-Dist: openai>=1.0; extra == "benchmark"
Requires-Dist: httpx>=0.23; extra == "benchmark"
Provides-Extra: dev
Requires-Dist: pytest>=7; extra == "dev"
Requires-Dist: moto[ec2]>=5; extra == "dev"
Requires-Dist: ruff==0.15.21; extra == "dev"
Requires-Dist: build>=1; extra == "dev"
Requires-Dist: twine>=5; extra == "dev"
Requires-Dist: openai>=1.0; extra == "dev"
Requires-Dist: httpx>=0.23; extra == "dev"
Dynamic: license-file

# gpuroutertest

The easiest way to run a private, GPU-backed LLM endpoint in your own AWS account — one command up, one command down.

## Prerequisites

- Python 3.9+
- AWS credentials (`aws configure`, `AWS_PROFILE`, or SSO) with permission to manage EC2
- GPU (`G`/`P`) instance quota in your region

## Install

```bash
pip install gpuroutertest
```

## Usage

```bash
gpuroutertest list-models                       # pick a <model_id>
gpuroutertest deploy <model_id> --size small --region us-east-1
```

Prints an endpoint URL and an API key once the model is loaded and healthy.

```bash
curl $ENDPOINT/chat/completions \
  -H "Authorization: Bearer $API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"<model_id>","messages":[{"role":"user","content":"Hello"}]}'
```

`<model_id>` is any id from `gpuroutertest list-models`; use that same id in `deploy` and in the `"model"` field.

Then tear it down to stop billing:

```bash
gpuroutertest delete $API_KEY --region us-east-1 -y
```

## Spot instances

Cheaper capacity behind a stable Elastic IP, auto-relaunched if AWS reclaims it. The endpoint URL never changes.

```bash
gpuroutertest deploy <model_id> --size small --region us-east-1 --spot
```

## Large models

Serve weights from a same-region public S3 bucket instead of HuggingFace — free, multi-GB/s, no flaky boot downloads.

```bash
gpuroutertest deploy <model_id> --size large --region us-east-1 \
  --model-uri s3://<bucket>/<model_id>
```

## Cost guard

Auto-terminate after N minutes:

```bash
gpuroutertest deploy <model_id> --size small --region us-east-1 --ttl 120
```

## Load testing

```bash
pip install gpuroutertest[benchmark]

gpuroutertest --test --endpoint $ENDPOINT --api-key $API_KEY \
  --model <model_id> --concurrency 10 --prompt-length 1k
```

## Python

Full lifecycle in a script — launch, call the endpoint, tear down:

```python
import gpuroutertest as gr
from openai import OpenAI  # pip install gpuroutertest[benchmark]

# 1. Launch (blocks until the model is loaded and healthy)
dep = gr.deploy("<model_id>", size="small", region="us-east-1")  # id from gr.list_models()
print(dep.endpoint_url, dep.api_key, "healthy:", dep.healthy)

# 2. Call it — the endpoint is OpenAI-compatible
client = OpenAI(base_url=dep.endpoint_url, api_key=dep.api_key)
resp = client.chat.completions.create(
    model=dep.model,
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

# 3. Tear down to stop billing
gr.destroy(dep.api_key, region="us-east-1")
```

`dep` exposes `endpoint_url`, `api_key`, `model`, `instance_id`, `public_ip`, `healthy`, and (for `--spot`) `elastic_ip` / `recovery_armed`. Any HTTP client works — swap the OpenAI SDK for `requests`/`httpx` and POST to `dep.endpoint_url + "/chat/completions"` with `Authorization: Bearer <api_key>`.

Deploy without blocking, then reconnect later by API key:

```python
dep = gr.deploy("<model_id>", size="small", region="us-east-1", wait=False)

info = gr.get_deployment(dep.api_key, region="us-east-1")  # state, endpoint, IP
for d in gr.list_deployments(region="us-east-1"):          # everything running
    print(d.api_key, d.state, d.model, d.endpoint_url)
```

## CLI

Every command takes `--profile/-p` and `--region/-r`.

```bash
gpuroutertest list-models                  # models you can deploy
gpuroutertest deploy   <model_id>          # launch an endpoint
gpuroutertest ps                           # your running deployments
gpuroutertest status   <api_key>           # state, capacity, health
gpuroutertest endpoint <api_key>           # just the URL
gpuroutertest logs     <api_key>           # instance boot logs
gpuroutertest delete   <api_key> -y        # terminate, stop billing
gpuroutertest version
```

### `deploy` flags

| Flag | Purpose |
|------|---------|
| `--size` | `small` / `medium` / `large` tier from the registry |
| `--hf-token` | access token for gated models (or set `HF_TOKEN`) |
| `--model-uri` | pull weights from a same-region public S3 prefix instead of HuggingFace |
| `--spot` | spot capacity behind a stable Elastic IP, with auto-relaunch on reclaim |
| `--cidr` | restrict who can reach port 8000 (default `0.0.0.0/0`) |
| `--ttl` | auto-terminate after N minutes |
| `--timeout` | minutes to wait for the health check (default 30) |
| `--no-wait` | return as soon as the instance is running |

### `--test` flags

| Flag | Purpose |
|------|---------|
| `--test[=client\|server\|both]` | run a load test, pick which side(s) to report (bare = both) |
| `--endpoint` / `--api-key` / `--model` | which server to hit |
| `--concurrency` / `-c` | requests kept in flight (default `1`) |
| `--prompt-length` / `-l` | input size tier: `100`, `1k`, `8k` (default `100`) |
| `--requests` | total requests to send |
| `--warmup` | throwaway requests before timing (default `3`) |
| `--metrics-url` | override the server `/metrics` URL |
| `--matrix` | sweep concurrency 1/10/100 × input 100/1k/8k |

## Errors and limits

- Port 8000 is open to `0.0.0.0/0` by default, protected only by the API key. Use `--cidr` in real use.
- Traffic is plain HTTP (no TLS). Put it behind a proxy for anything beyond dev.
- `--spot` recovers automatically but not instantly — there is a downtime gap while the replacement reloads the model.
- Container logs need SSH/SSM, which are intentionally not provisioned. `gpuroutertest logs` gives EC2 console output only.

---

# Details

## Models

Models live in [`sdk/gpuroutertest/registry.json`](sdk/gpuroutertest/registry.json), keyed by model id, each holding the per-size config (instance type, disk, server flags). That id is what you pass to `deploy`, what `list-models` prints, and what you send as `"model"` when calling the endpoint. Add a model by adding an entry — no code changes.

## Spot failover

`--spot` allocates an Elastic IP up front and serves the endpoint from it, so the URL survives instance replacement. At deploy time it also provisions two more pieces, all tagged with the deployment's API key so `delete` removes them together:

- an **EventBridge rule** that fires on the `EC2 Spot Instance Interruption Warning` event
- a per-deployment **Lambda** that the rule invokes

```text
Spot reclaim warning (~2 min notice)
        ↓
EventBridge rule  →  recovery Lambda
        ↓
Launch a replacement spot instance (same model, size, and boot config;
tries each AZ on capacity errors)
        ↓
Re-point the Elastic IP at the replacement  →  endpoint URL unchanged
```

The replacement is an exact clone of the original launch — same model, weights source, size, and vLLM flags. The rule isn't pinned to an instance id, so it keeps protecting each replacement in turn across any number of reclaims; the Lambda ignores warnings for instances outside this deployment.

**There is a downtime gap.** Loading a model into GPU memory takes several minutes — 8–15 for large models — far longer than the ~2-minute reclaim notice. Failover is *automatic recovery*, not *zero-downtime*: the endpoint is unreachable until the replacement finishes booting, then returns at the same URL. To ride through a reclaim with no gap, run two deployments behind your own load balancer. (Inference is stateless, so there is nothing to checkpoint.)

`--spot` needs permission for Elastic IPs, Lambda, EventBridge, and IAM (to create its recovery role). `status` shows `Capacity: spot (failover armed)`. `delete` removes the instance, Lambda, rule, and Elastic IP in one shot; the shared IAM role is left for reuse and costs nothing.

## Weights from S3

With `--model-uri`, the instance fetches weights from that S3 prefix with [s5cmd](https://github.com/peak/s5cmd) (`--no-sign-request`) instead of HuggingFace. Because the bucket is in-region the transfer is free and multi-GB/s, and vLLM serves the local copy under the original model id — callers address the endpoint identically. It's opt-in per deploy; for small models the default HuggingFace path is fine.

- The bucket must be in the **same region** as `--region` (cross-region would be slow and incur egress).
- The prefix must point at the folder directly containing `config.json`, the safetensors shards, `model.safetensors.index.json`, and the tokenizer files.
- The value must start with `s3://` (validated before any AWS call, so typos fail fast).

**Seeding a bucket** (one-time per model): [`scripts/seed_model_to_s3.py`](scripts/seed_model_to_s3.py) streams a HuggingFace repo straight into S3 without staging it on local disk, and is resumable. For an unattended overnight run use [`scripts/seed_overnight.sh`](scripts/seed_overnight.sh), which re-runs the seeder until every file is confirmed uploaded.

```bash
python scripts/seed_model_to_s3.py \
  --model  <model_id> \
  --bucket <bucket> \
  --region us-east-1
```

## Benchmark output

`--test` measures the same run from two vantage points: **client side** (what a caller experiences, including the network round-trip) and **server side** (read from vLLM's `/metrics`). The gap between them is network latency.

```
Model: <model_id>
Prompt Length: 1K
Concurrency: 10

                      Client        Server
Avg TTFT             1694 ms       1077 ms
P50 TTFT             1663 ms       1300 ms
P95 TTFT             2408 ms       2380 ms
TPM                     9757          9757

Network latency (client - server TTFT): 617 ms
(server observed 24 request(s) during the run)
```

- **TTFT** — time to first token; the responsiveness a caller feels.
- **TPM** — output tokens per minute; throughput.
- **Avg / P50 / P95** — average, median, and the slow 5%. P95 far above P50 means requests are queueing.

**Warmup matters.** A freshly-booted server pays a one-time cold-start cost of several seconds. `--warmup` fires throwaway requests before the clock starts so that cost never skews results.

Server-side numbers are best-effort: if `/metrics` is unreachable (e.g. blocked by a security group), the test falls back to client-side numbers with a warning. Test inputs come from [`sdk/gpuroutertest/prompts/`](sdk/gpuroutertest/prompts/) — one entry per `---`-separated block; edit them to use your own. Inputs repeat when `--requests` exceeds the number of unique ones.

`--matrix` sweeps the full grid (concurrency 1/10/100 × input 100/1k/8k, 9 runs), prints the table, and saves it to `inference_benchmark.txt`. A failing cell shows as an `ERROR` row rather than aborting the sweep.

```
Benchmark Report
Model: <model_id>
(TTFT/Network Latency in ms; TPM in tokens/min; both server-side, from vLLM. Network Latency = client - server TTFT)

Length    Conc    Avg TTFT    Srv TTFT           TPM   Network Latency  Fail
---------------------------------------------------------------------------
100          1         604         106          1415               499     0
100         10         690         184          8208               506     0
100        100         734         225         15857               509     0
1K          10        1715        1034          7413               681     0
8K          10        7406        6654          4924               751     0
```

## License

MIT — see [LICENSE](LICENSE).
