Metadata-Version: 2.4
Name: kimpton-evalrouter-sdk
Version: 0.1.1
Summary: Typed Kimpton evaluation client and command-line interface
License-Expression: LicenseRef-Kimpton-EvalRouter-SDK
License-File: LICENSE
Requires-Python: <3.14,>=3.12
Requires-Dist: httpx==0.28.1
Description-Content-Type: text/markdown

# EvalRouter CLI

Evaluate models from your terminal: discover benchmarks and model routes, review
a spending cap, run an evaluation, and export its results. Requires Python 3.12
or 3.13. Distributed under the proprietary [Kimpton EvalRouter SDK License](LICENSE).

## Install and create your account

```sh
python -m pip install --upgrade kimpton-evalrouter-sdk==0.1.1
evalrouter --help
```

The package supplies the `evalrouter` command; no repository checkout is needed.
[Create an EvalRouter account](https://evalrouter.ai/register), verify your email,
and create a workspace key in [API key settings](https://evalrouter.ai/settings/api-keys).
Package installation does not grant account access or evaluation credit.

Set `EVALROUTER_API_KEY` and `EVALROUTER_WORKSPACE_ID` through your secret manager
or private environment. Use `https://api.evalrouter.ai` for `EVALROUTER_BASE_URL`,
without `/v1`. Never put credentials in arguments, URLs, request files or logs.
The CLI uses an existing workspace key; browser sign-up is a separate step.

## Discover, quote, then run

```sh
evalrouter catalog --status active --json
evalrouter catalog --models --json
evalrouter catalog --slug BENCHMARK_SLUG --json
```

Choose a profile whose `quote_availability.status` is `ready_for_quote` and a
compatible managed model route from those responses. An active catalog entry
alone does not mean it is ready to run.
Save this as `quote.json`, replacing both uppercase identifiers:

```json
{
  "model": {"kind": "managed", "route_id": "MODEL_ROUTE_ID"},
  "selection": {"profile_ids": ["BENCHMARK_PROFILE_ID"]},
  "coverage": {"mode": "sample", "sample_count": 3, "seed": 42},
  "max_charge_microusd": "1000000"
}
```

```sh
evalrouter quote --config quote.json --json
```

Review compatibility, coverage, cost components, warnings and expiry. A quote
starts no paid work. The example's $1 platform cap is not a price guarantee or
promise that a particular evaluation fits. Paid runs require available credit.

Persist the returned quote ID and an operation key before submitting:

```sh
evalrouter run --quote REVIEWED_QUOTE_ID --idempotency-key SAVED_OPERATION_KEY --json
evalrouter wait RUN_ID --wait-timeout 3600 --json
evalrouter results RUN_ID --json
evalrouter export RUN_ID --format json --output result.json
evalrouter export RUN_ID --format csv --output result.csv
evalrouter export RUN_ID --format html --output result.html
```

Replace `RUN_ID` with the returned ID. A lost submission response is recovered
with the same quote and operation key; a new key may start separate work.
Inspect terminal status, coverage, errors and billing alongside scores. A sample
is not a full-benchmark score. Exports refuse overwrite by default; use the
result's integer `--version` for repeatable reports.

## Supported commands

`catalog` (`--models` or `--slug`), `connections list/create/check/update/disable`,
`quote`, `run`, `status`, `wait`, `cancel`, `results` and `export`.
Run each command with `--help` for exact flags. A connected model's credential
comes from `--key-env`, never a credential argument. Connection checks can send
a small model request, and your model provider may bill separately.

For `wait` and `run --wait`, `--progress auto` shows elapsed time and status in
an interactive terminal, with a percentage when validated processed and planned
counts are available. Use `--progress plain` for structured progress or
`--progress off` to suppress it. Redirected stderr and `--json` retain structured
progress. A progress percentage counts processed samples; it is not a score.

Use `--config -` for JSON on stdin. `--json` writes one result on stdout and
progress on stderr. Exit codes: 0 success; 1 API/transport or failed run;
2 invalid input or rejected request; 4 partial/cancelled waited run. Interrupting
a local wait does not cancel server work. Use `evalrouter cancel RUN_ID`
explicitly when cancellation is intended.

See the [CLI documentation](https://evalrouter.ai/developers/docs) for account
setup, model selection, spending semantics, recovery and versioned exports.
The CLI uses the [model API](https://evalrouter.ai/developers/docs/model-api),
which is also documented for direct HTTP integrations.
